2 part question on working with large 8bit bitmaps

hi PMH4514,
Thanks for your kind words about that code I wrote back in 2002... I'd forgotten all about it (and got a chuckle from the end :-)

The first part should be relatively simple.   In an 8-bit grayscale image, the pixels are exactly one byte each... so rather than
     int nSrcBytesPerLine= ((nSrcWide * 3)+3) & 0xfffffffc;  // divible by 4
I think it it would be simply:
     int nSrcBytesPerLine= (nSrcWide+3) & 0xfffffffc;  // divisible by 4 (round to dword)

And I think there may be some changes to parameters in CreateDibSection... I'll take a look at that.

As to the reverse operation, I see nothing in the documentation that appears to prohibit DIB Sections that are 160MB in size...  Does your system have that much RAM available?  If not, are you using a FileMapping handle as the "hSection" argument?   Or are you passing NULL?  I can image the latter would fail in a RAM-shortage.

Before I start to work on this, consider this (and please respond):
It may be smarter to not use a BMP file at all.  
If you are writing a specialty application to pan around in and display parts of a huge image, then there may be no compelling reason to use a BMP file to store the data.  

For instance, you could store the data into an SQL Server database as a series of 'blob' records.  Rather than seeking all over the disk to find the desired part of the monster BMP, you could make a quick indexed lookup in the database to get the desired chunk of image data directly -- you would then just feed it into the prepared DIB Section and it would be ready to display.  

Even if you want to use the filesystem directly (for speed or whatever) it might be simplest to just write the image data "raw" into a file so that you can avoid some complications with handling the BMP header, etc.

-- Dan

actually Dan I tried removing the multiple of *3 in that line you referenced (before I even posted this question)  - the results were odd to say the least. I got a squished, trippled version of what i expected (sorta part of the image I was looking for, repeated 3 times horizontally, occupying much less vertical space than I'd expect)

I am in fact writing a specialty app to pan around, and in fact I already do use a SQL database to manage the 1000x1000 single frames that are captured. Our users however end up taking the composites into other photo editing programs, and stitching them on their own, and it has been required of me to build this support in to the app, using bitmaps.

>>If not, are you using a FileMapping handle as the "hSection" argument?  
Again, my understanding of filemapping is still very limited. So no, I am not, please ellaborate if you think it prudent.

I will investigate further using additional RAM, but I prefer not to throw hardware at a software problem unless in fact it is simply a lack of RAM. I'll track down an extra chip when I'm back at work and see if it affects the dimensions at which the HBITMAP is returned NULL.

as to the tripled image...
I was forgetting that the DIB Sections are DIB... that is, they are in a device independant format, and in this case, they are going to use 3 bytes per pixel.  If we use CreateCompatibleDC(NULL) we are going to use a DC that is compatible with the screen -- rather than the grayscale bitmap and that could explain (... something, I'm not sure what  :-)

This is the kind of problem that I can solve when I have something concrete to work with.  Can you provide a URL to several of these grayscale bitmaps so that I can download them and try some experiments?

Incidently, I do believe that bumping your system up to, say, 2GB or 4GB of RAM will allow the code to load the 160MB bitmaps without breaking a sweat... but that is not a particularly satisfying solution to that part of this problem.

actually I can't legally share the actual images (they are research oriented) - but if you went into photoshop or Paint or whatever and created a 1000x1000 image of naything, saved as 8bit grayscale,  that could mimic one of the elements. The composite would simply be them stitched together into a large square (or rectangle)

I'll update you with results of the increased ram test.  It's not a satisfying solution, but certainly one we can work with (we have a limited user base for our software, and we provide the hardware as part of a full system, so we have complete control over the configuration)

The exact format of the bitmap really does make a difference and it makes it much more difficult to work on these things when having to guess, then work up an example, and finding that it works great on the bitmaps I made myself but not on the bitmaps you have...

A "grayscale image" can be a full-color image in which all of the "colors" are shades of gray and each pixel's data is a 16, 24, or 32-bit packet of data.  Or each pixel can be exactly 8 bits and each value is an index into a lookup table (the pallette).   So many functions convert the data from its on-file representation to a different in-memeory representation that one can easily be misled...

Execute this piece of code

void CBmpTestDlg::OnButton2()
{
      CString           sFile= "c:\\temp\\test_01.bmp";

      HANDLE            hFile;
      HANDLE            hFileMapping;
      BITMAPFILEHEADER* pBMFH;
      BITMAPINFO*       pBMI;    

      hFile=        CreateFile( sFile,GENERIC_READ|GENERIC_WRITE,0,0,OPEN_EXISTING,0,NULL);
      hFileMapping= CreateFileMapping( hFile,NULL,PAGE_READWRITE,0,0,0 );

      pBMFH= (BITMAPFILEHEADER*)MapViewOfFile( hFileMapping,FILE_MAP_ALL_ACCESS,0,0,0);  // entire file
      pBMI=  (BITMAPINFO*)     (pBMFH+1);
}

placing a breakpoint at the end bracket.  RUn the code and wHen it hits, use the QuickWatch to see what's in
       pBMI->bmiHeader
If biCount is 8 and biClrUsed is 256, then you have palletized data.  

Collect the value of biColors (it is an address in memory).  Use the memory viewer (Alt+6) to examine the data at that address.    Each four bytes represents an RGBx lookup value.  For instance, in the pixel data, if the value is 0x00 then the first four bytes (n+0 to n+3) in bmiColors are the RGBx for that pixel.  If the pixel value is 0x01, then bytes 4-7 are the RGBx for that pixel.

Examine the palette that way for several of your images.

In the other question, the code assumes non-palletized data -- that is, it assumes that each pixel is 32 bits and the pallette is not used.  That simplified things because the data for a DIBitmap Section in memory is the same as when on disk.

This is an important issue for your application, because if you "stitch together" pictures that have different pallettes, then everything will go to hell in a handbasket.  If you use the upcoming "SetImgSubset() fn and overwrite a piece of one bitmap with data from another, there is every chance that the shading of the data will be quite different -- it will look like a patchwork quilt.

If the pallette is always the same for all images, then that problem goes away.  Or if you use 32-bit color data, that problem also goes away.

-- Dan

oh right, I see what you're saying about the different types of 8bit images. My images are not using palletized data.

I can send you an example image that is "legal" for me (captured with the software, but not of any relevant subject matter)

Problem is, it seems all the free image hosting sites convert the format to JPG or PNG..  Do you have an address to which I can send it (or know of a free site that does not change the format?)

ALL images used in the stitch will be of exactly the same format.  I ran he bit of code you requested, the contents of bmiHeader are as follows:

biSize      40
biWidth    1000
biHeight    1000
biPlanes    1
biBitCount  8
bitCOmpression 0
biSizeImage  1000000
biXPelsPerMeter 0
biYPelsPerMeter 0
biClrlUsed  0
biClrlImportant 0

With regard to examining the colors, not sure what you're looking for from me. I pasted the address of biColors (0x07fe0036) into the memory viewer and the first 5 lines:

07FDFF28
07FDFF55
07FDFF82
07FDFFAF
07FDFFDC

are almost entirely populated with ??

Dan - I put two bitmap files in a zip file, uploaded to savefile.com for you (16MB)

http://www.savefile.com/files/4814674

It's an example, one 1000x1000 image and one 8000x8000 stitch that my software created.  (the biggest stitch I have to support, same format across the board, is 40,000 x 40,000

That is palettized data, but the pallette is uniform.  For instance, the RGB for a pixel with a value of 0x17 is 0x00171717;  a pixel with a value of 0xF3 is 0x00F3F3F3, and so forth  ... so it will be possible to read from and write to the file directly.  

It also allows a trick if we end up working with DIB Sections... we can use every fourth byte of color data in the in-memory pixel data and it will correspond to the pixel-data in the file.

I just got hit with a pile of stuff that I need to take care of immediately, but I'll be looking at this in the next few days and I'll have some code for you soon.

-- Dan

BTW, to look at the palette data, use this:

      DWORD* pRGB= (DWORD*)pBMI->bmiColors;

at the end of the previous code.  THen in the watch window use this expression:

     pRGB,256

That makes the debugger display it as DWORDs (which is all that an RGB Quad really is) and the debugger will display all 256 of them in a nice neat column.

thanks Dan. I'll look forward to your updates.

I wasn't able to secure additional RAM yet to test the creation of the 160MB bitmap either.

by the way:

"that is palettized data, but the pallette is uniform.  For instance, the RGB for a pixel with a value of 0x17 is 0x00171717;  a pixel with a value of 0xF3 is 0x00F3F3F3, and so forth  ... so it will be possible to read from and write to the file directly.  "

that's what I meant when I said "it's not palettized" -  my thought was that "palettized" would be when there was a lookup table (ie. the palette)

so when each pixel is a lookup into a palette, that is "non palettized" ?

I don't know if I am using the terminology correctly.  But if there is a pallette (and there *is* one in the image you sent me), then the system uses it when it generates a DIBitmap.  A Device Independant Bitmap has no palette at all -- each pixel fully describes itself.

Imagine if you wanted to enhance a grayscale image:  You could make certain ranges of the pixels darker or lighter just by modifying the pallette.  Or you could make a certain shade of gray into bright red... etc.  But in a DIB, you would have to cruise through the entire image, changing the pixels themselves.

gotcha!

looks very good Dan. I'll play with it. Question though - "who created" the file DestTest.bmp? In other words, how is the first stitch handled, when there is not currently any file to map?  You are using CreateFile on the destination, with OPEN_EXISTING, do I assume your dest bitmap already exists at the full size? Does a 160MB (or whatever size necessary) bitmap file have to be created before anything can be stitched into it and if so, how?

Currently I end up with my huge HBITMAP in memory, and then I'm using CDIBSectionLite::Save(path) to both create and write to the file.  (CDIBSection from CodeProject)

Yes, this code assumes that the destination BMP file exists and is the full size.

>> ...bitmap file have to be created before ... and if so, how?

On my system (732MB or RAM) I was able to use Ms PaintBrush to create a 15,000 x 15,000 8bpp bitmap file -- 215 MB.  Interestingly, PBrush was using about four times that amount of RAM + Virtual Memory while running -- so I think we can assume that it is using 32bpp data while running, then saving as 8bpp.  The save operation temporaily increased VM usage and took a long time and slowed down all other processes (virtually brought my PC to it's knees).

When I tried to create larger bitmaps, it failed with a "shortage of resources" message.

Your 40,000 x 40,000 bitmap will actually be 1.6 GB on disk (not 160 MB) and a general-purpose tool like PaintBrush would likely need about 7GB of physical RAM in order to hold the entire thing in memory for processing.  So PBrush.Exe is not going to do the job.   It's possible that there are specially graphics programs that do no require the entire thing to be held in memory.  If so, use that tool.

If you can't find a tool to do it, it would be possible to write a short program to create an empty 1.6 GB BMP file in the desired format.   If you can't find a tool to do it, ask another Q here at EE and I'll write you a small console app to do it.

Keep in mind that in the end, there may be no tool available to see the entire mamoth BMP file at once... you will need to treat it as a sort of pixel database and write custom code to pull in subsets for viewing and editing.

-- Dan

Dan -

>>1.6 GB on disk (not 160 MB)
oh yah, typo. sorry (big typo!)

>>Yes, this code assumes that the destination BMP file exists and is the full size.

hmm.. unfortunately, I have to create everything within my software and as I capture single frames, stitch them into position within a larger composite, that did not exist prior.  Unless perhaps I "shipped" the software with a default huge image of each possible size, sitting there waiting for me to copy it a new file prior to acquiring the elements, into which the stitched elements would be pushed. Sorta like "template" bitmap files that I would copy and then use.

Not a preferred solution

>>If you can't find a tool to do it, it would be possible to write a short program to create an empty 1.6 GB BMP file in the desired format.

This code I would be interested in seeing. Rather than a console program, a function which I would tell the size and path, and it would create the empty file of the appropriate size.  That in conjunction with your efforts here I believe would be suitable.  Go ahead and write that up if you don't mind, I'll then post the question (I'll make it worth 500pts for ya) and then you can immediately "answer" it :)

>>Keep in mind that in the end, there may be no tool available to see the entire mamoth BMP file at once.

Exactly where my need for GetImgSubset() came into play (our users can take the huge file and do with it what they like outside my program with whatever research/imaging tools they might have)  - but in so far as use of the file within my app, during the stitching, I also create a 250x250 version of the mammoth image which they can see and pan a little selector around (like in photoshop) - I map the rect of the selector to the 250x250 rect to determine the 1000x1000 rect wtihin the mammoth composite I need to display.

I have written the requested function that creates huge bitmaps.  

IMPORTANT CODE CHANGE To ABOVE FN...

In testing the new code, I found and corrected a small but important bug in BlitDirectToFile().   Change the code near the end of the fn to...

      for (int y=0; y<nPxHigh; y++ ) {
            int nCurLineDst= (nHighDst-1) - (y + nDstY);  // range is 0 to n-1

Here's the other question:

https://www.experts-exchange.com/questions/21794366/Need-function-to-create-very-large-empty-bitmap-file-for-use-with-FileMapping.html

Didn't get a chance to tryout your code yet, my USB ports have all stopped working (?@#$@!#) new drivers, windows updates, nothing has worked.. stopped me dead in my tracks!

Thanks for the points and the A.  
If you need help with modifing the GetImgSubset() fn for the 8bpp data, let me know.

-- Dan

>>If you need help with modifing the GetImgSubset()

oh thats right, the first part of the question! I do in fact, I wasn't able to get it working with the single change you mentioned in your first response. Then I got sidetracked having to setup a new dev box for myself.

See if you can get the GetImgSubset to work with that large composite image I sent you. I'm not sure but there may also be errors in the file handling itself, that is, the fn would be called as the mouse moves around - a new subset rect is calculated based on mouse movements, and the fn is called, so it could happen frequently. I did try moving the CFile variable into a class member, opening it once and closing it once, but still had errors. I'll try to collect some more specific details of the problems I had.  I thought about as well perhaps making a worker thread that would prepare the corresponding subsection for the next left/right/up/down movement, but didn't go that far..

//-----------------------------------------------------
// creates and returns an HBITMAP compatible with current display (assumes 24bpp data)
// by reading 8 bpp data from the file using data there to create 24bpp data
// AVOIDS READING ENTIRE SOURCE BITMAP into memory
//
//------------------------------------------------------
HBITMAP GetImgSubset_8bpp( CFile& cfBmp, CRect& rc )
{
                       //  only needs to be sizeof(BITMAPFILEHEADER) + sizeof(BITMAPINFO)
      BYTE buf[1000];  //  but handy for eyeball check
      cfBmp.Seek(0, CFile::begin);
      cfBmp.Read(buf, 1000); // Read header info TBD: check error

      BITMAPFILEHEADER* pBMFH= (BITMAPFILEHEADER*)buf;
      BITMAPINFO*       pBMI=  (BITMAPINFO*)     (pBMFH+1);    

      DWORD nBitsFileOffset= pBMFH->bfOffBits;  // first 8-bit pixel (x,y)=(0,0) is here

      int nSrcWide= pBMI->bmiHeader.biWidth;
      int nSrcHigh= pBMI->bmiHeader.biHeight;

      int nSrcBytesPerLine= (nSrcWide+1) & 0xfffffffe;  // divible by 2

      int nDstWide= rc.Width();
      int nDstHigh= rc.Height();

      //---------------------------------- create the destination bitmap
      HWND    hWnd=   AfxGetMainWnd()->m_hWnd;
      HDC     hDC=    ::GetDC( hWnd );

      HBITMAP hbmRet=CreateCompatibleBitmap( hDC, nDstWide,nDstHigh );
      //      BITMAP rBM;     // eyeball check
      //      GetObject( hbmRet, sizeof(BITMAP), &rBM );
      ASSERT( hbmDest != 0 );

    int nDstBytesPerLine= ((nDstWide * 4)+3) & 0xfffffffc;  // divible by 4
      int nDstSizeTotal= nDstBytesPerLine * nDstHigh;

      BYTE* pabDstBits=       new BYTE[ nDstSizeTotal ];    // buffer for 24bpp destination bitmap data
      BYTE* pabOneLineBuffer= new BYTE[ nSrcBytesPerLine ]; // buffer to read one line of 8bpp data from file

      BYTE* pDst= (BYTE*)pabDstBits;
      for (int y= rc.top; y<rc.bottom; y++ ) {
            int j= (nSrcHigh-1) - y;   // fixes upside-down-ness of file data
            int nFileOffset= nBitsFileOffset + (j*nSrcBytesPerLine);

            cfBmp.Seek( nFileOffset, CFile::begin );
            cfBmp.Read( pabOneLineBuffer, nSrcBytesPerLine );  // read one scanline of pixels from the file

            for ( int x=rc.left; x<rc.right; x++ ) {
                  pDst[0] = pabOneLineBuffer[x];   // each of R,B, & G is the same value for grayscale
                  pDst[1] = pabOneLineBuffer[x];
                  pDst[2] = pabOneLineBuffer[x];
                  pDst[3] = 0;

                  pDst += 4;
            }
      }

      LONG nCnt= SetBitmapBits( hbmRet, nDstSizeTotal, pabDstBits );
      ASSERT (nCnt== nDstSizeTotal);

      delete pabOneLineBuffer;
      delete pabDstBits;

      return( hbmRet );
}

excellent. Do you think for my purposes, when this will be called for each mousemove, that I should open the CFile once, and close it when done, or should it be opened and closed for each read?

I'd open it once -- in read-only mode -- and keep it open.  

However, that might be the least of your bottlenecks (unless the large bmp is on remote file server).  I think you'll end up with several optimizations such as moving in increments of 8 or 16 pixels, and/or caching a larger subset image for interactive scrolling (only back to going to the file when you near the edges)... that sort of thing.

-- Dan