IBM Content Manager client server Video streaming
Trying to find out the size of the data block used to store, retrieve and transmit video/audio data (stored as BLOB) from the Content Manager Serevr to the client.
Thank you.
Thank you.
ASKER
Hi Kent,
I am trying to understand a concept behind the storage, retrieval and transmission of large binary objects that represent audio and video data. I am sure this data is broken up into smaller chunks (I believe Oracle refers to them as blocks). The blocks are sequential, they can be retrieved and transmitted independently.
I gave 500 points for this question because it requires in-depth understanding of the internals, although the question is trivial. The knowledge deserves a reward :)
Thank you!
I am trying to understand a concept behind the storage, retrieval and transmission of large binary objects that represent audio and video data. I am sure this data is broken up into smaller chunks (I believe Oracle refers to them as blocks). The blocks are sequential, they can be retrieved and transmitted independently.
I gave 500 points for this question because it requires in-depth understanding of the internals, although the question is trivial. The knowledge deserves a reward :)
Thank you!
Hi Cyber,
BLOB data in DB2 works differently. As far as I know, the application will have to read the BLOCK object from the database and use it.
That said, I don't know of anything that would prevent a C/C++/Java application from doing its own blocking -- reading the data in chunks and using the chunks. The application could also, theoretically, act as a streaming interface and simply stream the data to a file/device as it is being streamed to the application.
But a BLOB object is treated as a single entity within DB2.
Kent
BLOB data in DB2 works differently. As far as I know, the application will have to read the BLOCK object from the database and use it.
That said, I don't know of anything that would prevent a C/C++/Java application from doing its own blocking -- reading the data in chunks and using the chunks. The application could also, theoretically, act as a streaming interface and simply stream the data to a file/device as it is being streamed to the application.
But a BLOB object is treated as a single entity within DB2.
Kent
ASKER
Kent,
I agree that a BLOB is a single entity. I am trying to figure out the organization of "sub-entities".
For example, when the BLOBs are transmitted over the network, they are transmitted in a sequence of network packets. Similarly, when they are retrieved from the hard drive, they are also NOT retrieved in 2GB chunks. Instead they BLOBs must be retrieved in some smaller data unit that are used by IBM Content Manager to store, retrieve and transmit the data.
Thank you for your help.
I agree that a BLOB is a single entity. I am trying to figure out the organization of "sub-entities".
For example, when the BLOBs are transmitted over the network, they are transmitted in a sequence of network packets. Similarly, when they are retrieved from the hard drive, they are also NOT retrieved in 2GB chunks. Instead they BLOBs must be retrieved in some smaller data unit that are used by IBM Content Manager to store, retrieve and transmit the data.
Thank you for your help.
Hi Cyber,
The network layer is completely independent. As far as TCP/IP is concerned, it was handed some data which is broken into chunks small enough to be transmitted on the network. That happens in the IP stack well below the application.
The BLOB data is stored in DB2 as a BLOB data object (up to 2GB). There is no built-in mechanism for accessing part of the BLOB item.
This page shows the table where Content Manager stores the BLOB objects. It's a single column in a table.
http://publib.boulder.ibm.com/infocenter/cmgmt/v8r3m0/index.jsp?topic=%2Fcom.ibm.apr.doc%2Frmblobs.htm
Kent
ASKER
Kent,
The link you provided points to the table of a Resourse Manager (not the Content Manager). Also, I believe, BLOB data is stored in DB tables only if the data is sufficiently small. Usually the DB tables store only the pointers, while the BLOB data resides elsewhere.
Here is a good read on point:
http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=/com.ibm.db2.luw.admin.dbobj.doc/doc/c0054525.html
Thank you.
The link you provided points to the table of a Resourse Manager (not the Content Manager). Also, I believe, BLOB data is stored in DB tables only if the data is sufficiently small. Usually the DB tables store only the pointers, while the BLOB data resides elsewhere.
Here is a good read on point:
http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=/com.ibm.db2.luw.admin.dbobj.doc/doc/c0054525.html
Thank you.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Kent,
Thank you for your help. I found the answer to my question on Page 75 of the pdf below:
http://www.redbooks.ibm.com/redbooks/pdfs/sg246410.pdf
IBM Content Manager uses Multimedia Filesystem that separate large media files into blocks. IBM Content Manager also uses "Wide Striping" - a feature that allows access of successive blocks of data from multiple disks and nodes.
For the benefit of the communiy I am pasting the content of the document above.
Isochronous file access
The Multimedia Filesystem uses real-time features of the AIX operating system
to prevent the kernel and other programs from interfering with isochronous data
delivery. It also implements real-time disk scheduling to execute disk I/O
operations in the proper order for achieving an uninterrupted flow of data to
clients.
Scalability
The Multimedia Filesystem provides a high degree of scalability, both in the
amount of data it can store and the data throughput it supports. The high
scalability is achieved by using large disk blocks and wide striping.
Large disk blocks: To efficiently support multimedia and supercomputing, a
file system must maximize throughput from the available disks. Since disk
throughput is strongly related to disk block size, the Multimedia Filesystem
uses a large disk block size, 256 KB being the default. Conventional file
systems optimize to reduce space rather than to increase throughput, by
using small disk blocks; for example, the AIX native file system uses 4 KB
blocks. Since the Multimedia Filesystem can store several small files (or
partial blocks at the end of large files) together in a single large block, its
disk-space utilization is comparable to that of a conventional, small-block file
system.
Wide striping: In both supercomputing and multimedia, much or all of the file
system activity is often directed at a single file. For example, many or all
clients of a Video-on-Demand system may be viewing the same movie
simultaneously the first time it becomes available. It is easy to image a
demand which is higher than the throughput of an individual disk (5-10 MB/s)
or even of an individual storage node in the SP2 (40 MB/s). Achieving higher
throughput than that of a disk or node from a single file requires striping it
across multiple disks and storage nodes, so that successive blocks of the file
go on different disks and nodes. Each Multimedia Filesystem can be striped
across as many as 512 disks. Since each file is striped evenly across all disks
in a stripe group, the disk load is inherently balanced, regardless of
file-access skew (the unevenness of demand for different files).
Thank you for your help. I found the answer to my question on Page 75 of the pdf below:
http://www.redbooks.ibm.com/redbooks/pdfs/sg246410.pdf
IBM Content Manager uses Multimedia Filesystem that separate large media files into blocks. IBM Content Manager also uses "Wide Striping" - a feature that allows access of successive blocks of data from multiple disks and nodes.
For the benefit of the communiy I am pasting the content of the document above.
Isochronous file access
The Multimedia Filesystem uses real-time features of the AIX operating system
to prevent the kernel and other programs from interfering with isochronous data
delivery. It also implements real-time disk scheduling to execute disk I/O
operations in the proper order for achieving an uninterrupted flow of data to
clients.
Scalability
The Multimedia Filesystem provides a high degree of scalability, both in the
amount of data it can store and the data throughput it supports. The high
scalability is achieved by using large disk blocks and wide striping.
Large disk blocks: To efficiently support multimedia and supercomputing, a
file system must maximize throughput from the available disks. Since disk
throughput is strongly related to disk block size, the Multimedia Filesystem
uses a large disk block size, 256 KB being the default. Conventional file
systems optimize to reduce space rather than to increase throughput, by
using small disk blocks; for example, the AIX native file system uses 4 KB
blocks. Since the Multimedia Filesystem can store several small files (or
partial blocks at the end of large files) together in a single large block, its
disk-space utilization is comparable to that of a conventional, small-block file
system.
Wide striping: In both supercomputing and multimedia, much or all of the file
system activity is often directed at a single file. For example, many or all
clients of a Video-on-Demand system may be viewing the same movie
simultaneously the first time it becomes available. It is easy to image a
demand which is higher than the throughput of an individual disk (5-10 MB/s)
or even of an individual storage node in the SP2 (40 MB/s). Achieving higher
throughput than that of a disk or node from a single file requires striping it
across multiple disks and storage nodes, so that successive blocks of the file
go on different disks and nodes. Each Multimedia Filesystem can be striped
across as many as 512 disks. Since each file is striped evenly across all disks
in a stripe group, the disk load is inherently balanced, regardless of
file-access skew (the unevenness of demand for different files).
There are quite a few items in that process that can be called blocks. Which one are you looking for?
Kent