marrowyung
asked on
The IBM flash system
Right now we found out some thing like flash system (see attached)
Please suggest any successful use of this kind of hardware on how it work well with all database server like DB2 V8.1, Oracle 10g/11g and 12c, MS SQL 2008 R2 with CU4, MySQL 5.5 Enterprise.
As you all know that MS SQL 2014 and Oracle 12c has some kind of in RAM database or table to operate, this kind of hardware seems has not much use by itself ?
Any topology on how to architecture this hardware in most of major company's architecture?
Flash-System.pdf
Please suggest any successful use of this kind of hardware on how it work well with all database server like DB2 V8.1, Oracle 10g/11g and 12c, MS SQL 2008 R2 with CU4, MySQL 5.5 Enterprise.
As you all know that MS SQL 2014 and Oracle 12c has some kind of in RAM database or table to operate, this kind of hardware seems has not much use by itself ?
Any topology on how to architecture this hardware in most of major company's architecture?
Flash-System.pdf
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Im a little confused, you state you understand what a in-memory data base is and you understand what SSD is and the differences. However you are still asking how to use a this device.
This device is nothing more than a SAN server that uses SSD/Flash Memory instead of spinning disks. You can use this anywhere you would use a SAN. You would use it anytime you want to reduce I/O wait time and thus improve I/O through-put.
You can use it as the data store for any db setup, however if you use it for a in-memory db you may not get any benefit.
This device is nothing more than a SAN server that uses SSD/Flash Memory instead of spinning disks. You can use this anywhere you would use a SAN. You would use it anytime you want to reduce I/O wait time and thus improve I/O through-put.
You can use it as the data store for any db setup, however if you use it for a in-memory db you may not get any benefit.
You said:
> Also the question still how to make use of this/worth to do ?
> on the dB architecture ?
The best -- and ONLY -- way to use the IBM device is as disk. Put your most often-accessed tables on it, I suppose. You will need to do some database profiling to figure out which tables those are, and how large an appliance you need to hold an entire table (plus, of course, room for growth over the next year, and room for expansion over the life of the appliance).
Again -- this appliance is *built* on RAM, but it cannot be used as RAM -- just like paper is made from trees, but you cannot use paper as a tree.
> > "Pick up a copy of the recommended HW configs for the in-memory
> > databases and see what they say. "
> what is that mean?
I imagine that SQL and Oracle and other databases have configuration guides that tell you how much RAM you need and how to configure it. If they do, note that the IBM appliance *cannot* be configured as RAM.
> Also the question still how to make use of this/worth to do ?
> on the dB architecture ?
The best -- and ONLY -- way to use the IBM device is as disk. Put your most often-accessed tables on it, I suppose. You will need to do some database profiling to figure out which tables those are, and how large an appliance you need to hold an entire table (plus, of course, room for growth over the next year, and room for expansion over the life of the appliance).
Again -- this appliance is *built* on RAM, but it cannot be used as RAM -- just like paper is made from trees, but you cannot use paper as a tree.
> > "Pick up a copy of the recommended HW configs for the in-memory
> > databases and see what they say. "
> what is that mean?
I imagine that SQL and Oracle and other databases have configuration guides that tell you how much RAM you need and how to configure it. If they do, note that the IBM appliance *cannot* be configured as RAM.
ASKER
giltjr,
"however if you use it for a in-memory db you may not get any benefit. "
yes, that's why I said if in memory is used (Oralce 12c / SQL server 2014), this one seems not much use.
let's requote from my first ticket.
"As you all know that MS SQL 2014 and Oracle 12c has some kind of in RAM database or table to operate, this kind of hardware seems has not much use by itself ?"
now :
"This device is nothing more than a SAN server that uses SSD/Flash Memory instead of spinning disks. You can use this anywhere you would use a SAN. You would use it anytime you want to reduce I/O wait time and thus improve I/O through-put. "
What I understand is this kind of hardware usally place in front of the DB structure and let all data read/write to operation on this tier first before the finally write to the spin disk. Therefore speed up a lot of read/write.
SelfGovern,
"The best -- and ONLY -- way to use the IBM device is as disk. "
this means use this one as the new SAN use SSD? where the SQL server write data to ?
"Again -- this appliance is *built* on RAM,"
as it use SSD and we use it for fast READ access? you know SSD is not good for write operation, right?
Then it will be good for reporting server so no one will complain the speed of reporting any more?
"I imagine that SQL and Oracle and other databases have configuration guides that tell you how much RAM you need and how to configure it. If they do, note that the IBM appliance *cannot* be configured as RAM. "
so as we keep pointing out, this can only used a the SQL\Oracle disk for data, right?
"however if you use it for a in-memory db you may not get any benefit. "
yes, that's why I said if in memory is used (Oralce 12c / SQL server 2014), this one seems not much use.
let's requote from my first ticket.
"As you all know that MS SQL 2014 and Oracle 12c has some kind of in RAM database or table to operate, this kind of hardware seems has not much use by itself ?"
now :
"This device is nothing more than a SAN server that uses SSD/Flash Memory instead of spinning disks. You can use this anywhere you would use a SAN. You would use it anytime you want to reduce I/O wait time and thus improve I/O through-put. "
What I understand is this kind of hardware usally place in front of the DB structure and let all data read/write to operation on this tier first before the finally write to the spin disk. Therefore speed up a lot of read/write.
SelfGovern,
"The best -- and ONLY -- way to use the IBM device is as disk. "
this means use this one as the new SAN use SSD? where the SQL server write data to ?
"Again -- this appliance is *built* on RAM,"
as it use SSD and we use it for fast READ access? you know SSD is not good for write operation, right?
Then it will be good for reporting server so no one will complain the speed of reporting any more?
"I imagine that SQL and Oracle and other databases have configuration guides that tell you how much RAM you need and how to configure it. If they do, note that the IBM appliance *cannot* be configured as RAM. "
so as we keep pointing out, this can only used a the SQL\Oracle disk for data, right?
--> so as we keep pointing out, this can only used a the SQL\Oracle disk for data, right?
Simple and only answer, it can only be use for disk data, period. Does not what type of files it is, you can store MS Word docs, PDF's, web pages, swap files, ect.
To make this as simple as possible, as you seem to be over thinking this a little, this device is nothing more than "a big super fast disk". So anyplace you would use disk device and you want lower latency for reads and write, you can use this device.
Simple and only answer, it can only be use for disk data, period. Does not what type of files it is, you can store MS Word docs, PDF's, web pages, swap files, ect.
To make this as simple as possible, as you seem to be over thinking this a little, this device is nothing more than "a big super fast disk". So anyplace you would use disk device and you want lower latency for reads and write, you can use this device.
ASKER
"Simple and only answer, it can only be use for disk data, period. Does not what type of files it is, you can store MS Word docs, PDF's, web pages, swap files, ect."
So we just use it on the disk other than the system one as it should only be attached to any running existing SQL\Oracle server, right?
" this device is nothing more than "a big super fast disk". So anyplace you would use disk device and you want lower latency for reads and write, you can use this device. "
yes, if you check the web, you will see that if this storage has 10TB, after RAID 5 that, we can still use the 10TB it offer, is it possible ?
So we just use it on the disk other than the system one as it should only be attached to any running existing SQL\Oracle server, right?
" this device is nothing more than "a big super fast disk". So anyplace you would use disk device and you want lower latency for reads and write, you can use this device. "
yes, if you check the web, you will see that if this storage has 10TB, after RAID 5 that, we can still use the 10TB it offer, is it possible ?
--> So we just use it on the disk other than the system one as it should only be attached to any running existing SQL\Oracle server, right?
It can be attached to ANY system. They do not need to be running SQL\Oracle. You could connect to a Windows or *inx system that is setup to be a file server.
--> yes, if you check the web, you will see that if this storage has 10TB, after RAID 5 that, we can still use the 10TB it offer, is it possible ?
Which model? There are 4 models, some with as little as 4.7 TB of usable storage and some with as much as 24.7TB.
It can be attached to ANY system. They do not need to be running SQL\Oracle. You could connect to a Windows or *inx system that is setup to be a file server.
--> yes, if you check the web, you will see that if this storage has 10TB, after RAID 5 that, we can still use the 10TB it offer, is it possible ?
Which model? There are 4 models, some with as little as 4.7 TB of usable storage and some with as much as 24.7TB.
nswer to your questions to me -- yes, use it for disk for your databases, particularly on tables that are read-intensive.
I can't guarantee that none of your users will complain about the speed of running reports -- but it will be less likely they'll complain if you're running off of tuned SSD, than if you're running off of typical hard drives.
I can't guarantee that none of your users will complain about the speed of running reports -- but it will be less likely they'll complain if you're running off of tuned SSD, than if you're running off of typical hard drives.
ASKER
giltjr,
"Which model? There are 4 models, some with as little as 4.7 TB of usable storage and some with as much as 24.7TB"
I forget it, it seems 720 system. but it doesn't matter, right? I am talking about the RAID level..
So just attach that IBM flashsystem and mount it as a disk volumn, for example, E:\, that's it?
SelfGovern,
"-- but it will be less likely they'll complain if you're running off of tuned SSD, than if you're running off of typical hard drives. "
yes.
So as a result, anything evident we can see that it don't support any MS SQL\Oracle\MySQL\DB2?
"Which model? There are 4 models, some with as little as 4.7 TB of usable storage and some with as much as 24.7TB"
I forget it, it seems 720 system. but it doesn't matter, right? I am talking about the RAID level..
So just attach that IBM flashsystem and mount it as a disk volumn, for example, E:\, that's it?
SelfGovern,
"-- but it will be less likely they'll complain if you're running off of tuned SSD, than if you're running off of typical hard drives. "
yes.
So as a result, anything evident we can see that it don't support any MS SQL\Oracle\MySQL\DB2?
Unless your application only runs on certain disks (unlikely), it will run on this.
Make sure you size the unit you buy correctly so that it can handle projected data growth (you don't want to buy a 5TB unit now that can't be expanded, only to have your 4.5TB database fill it up in three months.
Make sure you size the unit you buy correctly so that it can handle projected data growth (you don't want to buy a 5TB unit now that can't be expanded, only to have your 4.5TB database fill it up in three months.
--> So just attach that IBM flashsystem and mount it as a disk volumn, for example, E:\, that's it?
Not really. This is a SAN. It will most likely come with special drivers that you need to install. Once you install the drivers then you can create a mount point for it.
In today's world I can't think of any DBMS that is written to work with a specific hard drive. Especially since most large scale systems don't have hard drives directly mounted in them, they all use SAN's or even NAS's and any application must be able to support file systems on those.
Since you seem to be looking at one of these, the 1st thing you need to do is figure out what your current I/O performance (I/O's per second and MB's read and write) is and what your I/O trends are (read to write ratio). Then see if this device, or something like it, will actually improve your performance.
Not really. This is a SAN. It will most likely come with special drivers that you need to install. Once you install the drivers then you can create a mount point for it.
In today's world I can't think of any DBMS that is written to work with a specific hard drive. Especially since most large scale systems don't have hard drives directly mounted in them, they all use SAN's or even NAS's and any application must be able to support file systems on those.
Since you seem to be looking at one of these, the 1st thing you need to do is figure out what your current I/O performance (I/O's per second and MB's read and write) is and what your I/O trends are (read to write ratio). Then see if this device, or something like it, will actually improve your performance.
ASKER
giltjr,
"Not really. This is a SAN. It will most likely come with special drivers that you need to install. Once you install the drivers then you can create a mount point for it.
"'
yes you are right and once it attached, Windows should be able to detect it and find drive ?
"Since you seem to be looking at one of these, the 1st thing you need to do is figure out what your current I/O performance (I/O's per second and MB's read and write) is and what your I/O trends are (read to write ratio). "
you mean if that hardware meet the SQL requirement first ? it will jsut say latency less than 8 ms!!]
usually IBM disk has a very hight output on disk ! I don't worry about that much. FOr DB, I think IBM can be one of the best ! take a look on the infinitband hardware they got, speed can be 20GB/sec, much better than the fiber channel !
"Not really. This is a SAN. It will most likely come with special drivers that you need to install. Once you install the drivers then you can create a mount point for it.
"'
yes you are right and once it attached, Windows should be able to detect it and find drive ?
"Since you seem to be looking at one of these, the 1st thing you need to do is figure out what your current I/O performance (I/O's per second and MB's read and write) is and what your I/O trends are (read to write ratio). "
you mean if that hardware meet the SQL requirement first ? it will jsut say latency less than 8 ms!!]
usually IBM disk has a very hight output on disk ! I don't worry about that much. FOr DB, I think IBM can be one of the best ! take a look on the infinitband hardware they got, speed can be 20GB/sec, much better than the fiber channel !
Are you looking at this for an existing database/application?
Or is this a brand new setup?
Or is this a brand new setup?
ASKER
"Are you looking at this for an existing database/application?"
existing one! we will have DB2, MySQL, MSQL and Oracle.
if this is a drive only than it will be easy, or anyting else?
existing one! we will have DB2, MySQL, MSQL and Oracle.
if this is a drive only than it will be easy, or anyting else?
It should be fairly easy if you are experienced with SAN's. To the server, and the DBMS's, it will look like any other SAN mounted disk. If you are not experienced with SAN, they introduce concepts that might be difficult for you to grasp at first.
Not sure where you got 20GB/s for Infinaband. The specs for the 720 say read through-put is 3.3GB for FC and 5 GB for Infiniband and slightly less for write.
There are two storage capacities for the 720, usable after RAID 5 overhead of 5.2 and 10.3 TB.
Since you mentioned 4 different DBMS's I am assuming you are running them on four different servers, which means you will need either a FC or Infiniband switch. I would expect that Infiniband would be more expensive.
Not sure where you got 20GB/s for Infinaband. The specs for the 720 say read through-put is 3.3GB for FC and 5 GB for Infiniband and slightly less for write.
There are two storage capacities for the 720, usable after RAID 5 overhead of 5.2 and 10.3 TB.
Since you mentioned 4 different DBMS's I am assuming you are running them on four different servers, which means you will need either a FC or Infiniband switch. I would expect that Infiniband would be more expensive.
ASKER
"It should be fairly easy if you are experienced with SAN's. To the server, and the DBMS's, it will look like any other SAN mounted disk. If you are not experienced with SAN, they introduce concepts that might be difficult for you to grasp at first."
yeah, how about refer that one as the NAS, as long as we can find that out and mount to it, we can write data .
"Not sure where you got 20GB/s for Infinaband. The specs for the 720 say read through-put is 3.3GB for FC and 5 GB for Infiniband and slightly less for write. "
http://www-03.ibm.com/systems/au/storage/flash/720-820/specifications.html
4 x 8 Gb/s Fibre Channel
4 x 40 Gb/s QDR InfiniBand
"Since you mentioned 4 different DBMS's I am assuming you are running them on four different servers, which means you will need either a FC or Infiniband switch."
Yes, 4 x different servers, but I think it will be the same for all DB as you all keep saying it will be the same as SAN,assign a LUN and then assing a volumn, right?
"I would expect that Infiniband would be more expensive. "
I think so.. I need to contact the IBM sales on this .
yeah, how about refer that one as the NAS, as long as we can find that out and mount to it, we can write data .
"Not sure where you got 20GB/s for Infinaband. The specs for the 720 say read through-put is 3.3GB for FC and 5 GB for Infiniband and slightly less for write. "
http://www-03.ibm.com/systems/au/storage/flash/720-820/specifications.html
4 x 8 Gb/s Fibre Channel
4 x 40 Gb/s QDR InfiniBand
"Since you mentioned 4 different DBMS's I am assuming you are running them on four different servers, which means you will need either a FC or Infiniband switch."
Yes, 4 x different servers, but I think it will be the same for all DB as you all keep saying it will be the same as SAN,assign a LUN and then assing a volumn, right?
"I would expect that Infiniband would be more expensive. "
I think so.. I need to contact the IBM sales on this .
"4 x 8 Gb/s Fibre Channel
4 x 40 Gb/s QDR InfiniBand"
That is the raw speed and the number of the interfaces, not necessarily the actual though-put. The same doc that shows how many interfaces and at what speeds should show the actually read and write through-puts that I quoted.
If you have FC/InfiniBand switches it will be a single SAN, you setup your LUN(s) and then assign volumes.
However each server will need to have its own volume. I don't think you can shares volumes between servers and even if you could, I don't think it would be a good idea as that would impact performance. In fact for best performance you will probably what to have unique LUN's for each server.
4 x 40 Gb/s QDR InfiniBand"
That is the raw speed and the number of the interfaces, not necessarily the actual though-put. The same doc that shows how many interfaces and at what speeds should show the actually read and write through-puts that I quoted.
If you have FC/InfiniBand switches it will be a single SAN, you setup your LUN(s) and then assign volumes.
However each server will need to have its own volume. I don't think you can shares volumes between servers and even if you could, I don't think it would be a good idea as that would impact performance. In fact for best performance you will probably what to have unique LUN's for each server.
ASKER
"That is the raw speed and the number of the interfaces, not necessarily the actual though-put"
I think so !
"FC/InfiniBand switches it will be a single SAN, you setup your LUN(s) and then assign volumes."
any statment print that out ?
only to the same server you mean ?
"I don't think it would be a good idea as that would impact performance. "
Yeah, share SAN not good too ! tried that before, dedicated SAN always good, expecially for SQL server.
I think so !
"FC/InfiniBand switches it will be a single SAN, you setup your LUN(s) and then assign volumes."
any statment print that out ?
only to the same server you mean ?
"I don't think it would be a good idea as that would impact performance. "
Yeah, share SAN not good too ! tried that before, dedicated SAN always good, expecially for SQL server.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
"To put it as simple as possible. This is a SAN just basically like any other SAN. You would treat it just like you would any other SAN. The difference is it uses Flash Memory instead of spinning disks to store the data. Flash Memory gives you lower latency and MUCH faster access time when doing random reads."
excellent, we ringed up IBM to demo it for us !
"FC vs. Infinaband is up to you and your pocket book. If you already have a FC infrastructure then I would go with it. If you don't have FC, then price out FC vs. Infinaband and go with your pocket book. According to the doc, there is a 1.2GB/s difference in performance, Infinaband being faster. Only you would know if that would make a difference in your environment."
yeah, but Infinaband is IBM's own product, right?
"You don't need to dedicate a SAN to a single server. You can share a SAN between multiple/many servers. "
But for DB, dedicated or direct FB connection should be the best for DB, right?
"You can't share a single logical volume. That will cause serious problems. Shareing aYou will want at least two interfaces on the SAN if sharing, in fact having two interfaces no matter what would be best. That way if one I/O path fails you have an alternate path. Of course each server would need to have multiple interfaces also. "
nice.
excellent, we ringed up IBM to demo it for us !
"FC vs. Infinaband is up to you and your pocket book. If you already have a FC infrastructure then I would go with it. If you don't have FC, then price out FC vs. Infinaband and go with your pocket book. According to the doc, there is a 1.2GB/s difference in performance, Infinaband being faster. Only you would know if that would make a difference in your environment."
yeah, but Infinaband is IBM's own product, right?
"You don't need to dedicate a SAN to a single server. You can share a SAN between multiple/many servers. "
But for DB, dedicated or direct FB connection should be the best for DB, right?
"You can't share a single logical volume. That will cause serious problems. Shareing aYou will want at least two interfaces on the SAN if sharing, in fact having two interfaces no matter what would be best. That way if one I/O path fails you have an alternate path. Of course each server would need to have multiple interfaces also. "
nice.
--> yeah, but Infinaband is IBM's own product, right?
No. Infinaband is a industry standard.
http://en.wikipedia.org/wiki/InfiniBand
--> But for DB, dedicated or direct FB connection should be the best for DB, right?
The simple answer is "it depends." To me using a switch would be best for the future. Going through a switch allows you to add or swap out physical servers a little easier. With a switch you can connect a second server to the switch.
Properly setup you can have multiple servers accessing mounts on the SAN without drastically impacting each other. They can access different volumes over different interfaces. As long as the SAN's CPU can keep up, then there is no impact.
No. Infinaband is a industry standard.
http://en.wikipedia.org/wiki/InfiniBand
--> But for DB, dedicated or direct FB connection should be the best for DB, right?
The simple answer is "it depends." To me using a switch would be best for the future. Going through a switch allows you to add or swap out physical servers a little easier. With a switch you can connect a second server to the switch.
Properly setup you can have multiple servers accessing mounts on the SAN without drastically impacting each other. They can access different volumes over different interfaces. As long as the SAN's CPU can keep up, then there is no impact.
ASKER
"The simple answer is "it depends." To me using a switch would be best for the future. Going through a switch allows you to add or swap out physical servers a little easier. "
you are right.
:Properly setup you can have multiple servers accessing mounts on the SAN without drastically impacting each other. They can access different volumes over different interfaces. As long as the SAN's CPU can keep up, then there is no impact"
last time check about the Shared SAN, the latency is about 120 and the highest latency we can accept for SQL server is 8ms, huge lag behind. They all has SAN switch, that's why a lot of application share the same SAN.
result is not very good.
you are right.
:Properly setup you can have multiple servers accessing mounts on the SAN without drastically impacting each other. They can access different volumes over different interfaces. As long as the SAN's CPU can keep up, then there is no impact"
last time check about the Shared SAN, the latency is about 120 and the highest latency we can accept for SQL server is 8ms, huge lag behind. They all has SAN switch, that's why a lot of application share the same SAN.
result is not very good.
It depends on your SAN and how you are sharing your SAN switch. We have had SAN HUBS and SAN switches.
I really don't see a SAN switch adding a TON of overhead, so I would need to know a lot of information as to figure out why you were seeing 120ms. My initial guess is an older SAN switch, and older SAN, and a bad setup on the SAN.
We had one SAN that we were seeing in the 50ms range, however the setup on the SAN was terrible. The SAN was purchased with 2 shelves to start with and new shelves were added one at a time. The arrays were NOT redone each time a new shelf was added. So each new array was on the same shelf. This was not good. We took the time to rebuild some of the arrays so they were spread across shelves and latency went down to 5ms for those arrays. This was a shared SAN.
I really don't see a SAN switch adding a TON of overhead, so I would need to know a lot of information as to figure out why you were seeing 120ms. My initial guess is an older SAN switch, and older SAN, and a bad setup on the SAN.
We had one SAN that we were seeing in the 50ms range, however the setup on the SAN was terrible. The SAN was purchased with 2 shelves to start with and new shelves were added one at a time. The arrays were NOT redone each time a new shelf was added. So each new array was on the same shelf. This was not good. We took the time to rebuild some of the arrays so they were spread across shelves and latency went down to 5ms for those arrays. This was a shared SAN.
ASKER
"I really don't see a SAN switch adding a TON of overhead, so I would need to know a lot of information as to figure out why you were seeing 120ms. My initial guess is an older SAN switch, and older SAN, and a bad setup on the SAN. "
ok, can be the reason but probably not ! probably the one who setup the SAN switch accidently make this up / he didn't know how !
ok, can be the reason but probably not ! probably the one who setup the SAN switch accidently make this up / he didn't know how !
ASKER
When you're talking about in-RAM databases, you're talking about databases that are tuned to run in memory on servers that have lots of physical RAM available (that is RAM as RAM, not RAM-presented-as-disk)."
I am not confused, but in-RAM database is a built-in feature of Enterprise edition of Oracle and MS SQL now, we can use this one if we have a lot of RAM and we don't evne need to chanage our topology on this. Quite easy to go.
"Using SSD instead of disk will make most any database and other applications (CAD, programming, etc.) run faster without having to modify the application, just because reads from and writes to disk happen so much quicker. The in-memory databases are re-written code designed to take advantage of systems with huge amounts of memory, and are re-coded applications."
Yeah, we already do this on one of the SQL server 2008 R2 already, it running quite well.
"I'm guessing it would be a poor choice to run an in-memory DB on a system with a lot of SSD but not a lot of RAM "
Agree, we are not going to do this except we have > 256GB/512GB of RAM and we proof that SQL hasn't use up all of them, and we move table by table to the RAM.
"it will wear out your SSDs quickly, and it will be awfully expensive for little or no benefit.
"
yeah we know, even with Data provisioning.
"Pick up a copy of the recommended HW configs for the in-memory databases and see what they say. "
what is that mena?
Also the question still how to make use of this/worth to do ? on the dB architecture ?