[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1279
  • Last Modified:

Problem with caching (Cache dependency)

I've recently tried using caching on a small application I'm running.  Whilst it works fine (Saving loads of processor / load time), it seems to be generating a couple of exceptions every two or so hours.  I'm guessing this is when the cache expires or the application ends and it goes off to look for a new version.  Here's the exception that I'm getting:

System.NullReferenceException: Object reference not set to an instance of an object.
   at ProgramName.Global.RefreshCache(String key, Object item, CacheItemRemovedReason reason)
   at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)

The code I am using follows:

    'Page load event from the page that uses the data
   
        Private Sub Page_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        ' Make sure Cache item isn't empty, if it is, fill it up
        If Not Page.IsPostBack Then
            If Cache("dsMainExcelData") Is Nothing Or Cache("dsDropExcelData") Is Nothing Then
                Global.RefreshCache(Nothing, Nothing, 0)
            End If

            Dim dsMainExcelData, dsDropExcelData As DataSet
            dsMainExcelData = CType(Cache("dsMainExcelData"), DataSet)
            dsDropExcelData = CType(Cache("dsDropExcelData"), DataSet)

                  'Populates a datagrid with data
            dgExcelData.DataSource = dsMainExcelData.Tables(0).DefaultView
            dgExcelData.DataBind()

                  'Populates a dropdown list with data
            ddlProducts.DataSource = dsDropExcelData.Tables(0).DefaultView
            ddlProducts.DataBind()
        End If
          End Sub


    'From global.asax.vb
   
      Sub Application_Start(ByVal sender As Object, ByVal e As EventArgs)
            Dim onRemove As CacheItemRemovedCallback = New CacheItemRemovedCallback(AddressOf RefreshCache)
            Dim depFile As String = System.Configuration.ConfigurationSettings.AppSettings("dependencyFile").ToString()

            System.Web.HttpContext.Current.Cache.Insert("dsMainExcelData", GetMainExcelData(), New CacheDependency(depFile), _
                               Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, _
                               CacheItemPriority.High, onRemove)

            System.Web.HttpContext.Current.Cache.Insert("dsDropExcelData", GetDropExcelData(), New CacheDependency(depFile), _
                         Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, _
                         CacheItemPriority.High, onRemove)
      End Sub
      
      'GetMainExcelData and GetDropExcelData both return datasets
      
      Shared Sub RefreshCache(ByVal key As String, ByVal item As Object, ByVal reason As System.Web.Caching.CacheItemRemovedReason)
            Dim onRemove As CacheItemRemovedCallback = New CacheItemRemovedCallback(AddressOf RefreshCache)
            Dim depFile As String = System.Configuration.ConfigurationSettings.AppSettings("dependencyFile").ToString()

            System.Web.HttpContext.Current.Cache.Insert("dsMainExcelData", GetMainExcelData(), New CacheDependency(depFile), _
                               Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, _
                               CacheItemPriority.High, onRemove)

            System.Web.HttpContext.Current.Cache.Insert("dsDropExcelData", GetDropExcelData(), New CacheDependency(depFile), _
                         Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, _
                         CacheItemPriority.High, onRemove)

      End Sub

Any suggestions would be great.  Also, is this the right way to go about handling two cache items that rely on the same file?
0
Psychotext
Asked:
Psychotext
  • 9
  • 3
1 Solution
 
der_jthCommented:
The stack trace shows the null reference is coming from the RefreshCache method. On a quick look, there's only one line that seems to be a likely culprit:

Dim depFile As String = System.Configuration.ConfigurationSettings.AppSettings("dependencyFile").ToString()

It sounds so odd that it might be worth running the web app in debug mode (getting the source line numbers into the stack traces) to confirm this is really where it's coming from. I'm guessing something like this could happen on the application shutdown, but I've never heard of anything to this effect. If everything works just fine and you only need to get rid of the exceptions, you might want to check for non-nothingness of the AppSettings entry before calling ToString on it. Anyway, if you can get the line  number that causes the exception, that would probably help in further tracking down the issue.


Some other notes which might also help:

It might be better to have Application_Start call RefreshCache - now you've pasted the same code into those two methods. Also, your approach of always reloading the files after caching can cause side effects when your system is low on memory. For example, the cache can purge your Excel files to conserve memory, but then they're immediately reloaded. Well, if those two items of data are crucial to all the parts of your application, this is probably just fine - at this point the application would likely die anyway. But if they're only used pretty rarely, you might consider an alternative loading strategy: Get rid of the reloading callback (and the loading on application startup), and whenever you need the files, call a global method that retrieves them from the cache, and if they're not available, calls RefreshCache to load and cache them. This way the datasets won't consume memory unless you actually need them.
0
 
PsychotextAuthor Commented:
Thanks, I'll have a look into your suggestions and post back.
0
 
PsychotextAuthor Commented:
Ok... looking into it now (Busy couple of weeks!), changed to debug mode here's what is returned:

System.NullReferenceException: Object reference not set to an instance of an object.
   at ProgramName.Global.RefreshCache(String key, Object item, CacheItemRemovedReason reason) in Global.asax.vb:line 67
   at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)

This seems to be this line:

      System.Web.HttpContext.Current.Cache.Insert("dsMainExcelData", GetMainExcelData(), New CacheDependency(depFile), _
                         Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, _
                         CacheItemPriority.High, onRemove)

I have included some other debug code and here's what appears to be the flow of the program:

      08/12/2004 11:23:08 - Application Start
      08/12/2004 11:23:08 - Cache Insert - Main Excel
      08/12/2004 11:23:08 - Cache Insert - Drop Excel
      08/12/2004 11:44:49 - Error - line 67 (Line in post above)
      08/12/2004 11:44:49 - Error - line 67 (Line in post above)
      08/12/2004 11:44:49 - Application End
      
From what I can see, this error only occurs on application end. Still not sure what's causing it though.  I'm going to change my code a little to see if I can get to the bottom of this using other methods.  I'm also going to change it so that the depfile is setup in the cache insert string which may help.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
PsychotextAuthor Commented:
Found and fixed.  Updated code below:

      'Page load event from the page that uses the data

            Private Sub Page_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
                  If Not Page.IsPostBack Then

                        ' Make sure Cache items aren't empty, if either are, fill them up
                        If Cache("dsMainExcelData") Is Nothing Then
                              Global.RefreshCache("dsMainExcelData")
                        End If

                        If Cache("dsDropExcelData") Is Nothing Then
                              Global.RefreshCache("dsDropExcelData")
                        End If

                        Dim dsMainExcelData, dsDropExcelData As DataSet
                        dsMainExcelData = CType(Cache("dsMainExcelData"), DataSet)
                        dsDropExcelData = CType(Cache("dsDropExcelData"), DataSet)

                        'Populates a datagrid with data
                        dgExcelData.DataSource = dsMainExcelData.Tables(0).DefaultView
                        dgExcelData.DataBind()

                        'Populates a dropdown list with data
                        ddlProducts.DataSource = dsDropExcelData.Tables(0).DefaultView
                        ddlProducts.DataBind()
                  End If
            End Sub

      'From global.asax.vb

            Sub Application_Start(ByVal sender As Object, ByVal e As EventArgs)
                  RefreshCache("dsMainExcelData")
                  RefreshCache("dsDropExcelData")
            End Sub

            Shared Sub RefreshCache(ByVal strObjectName As String)
              System.Web.HttpContext.Current.Cache.Insert(strObjectName, GetExcelData(strObjectName), _
                    New CacheDependency(System.Configuration.ConfigurationSettings.AppSettings("dependencyFile").ToString()), _
                    Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, CacheItemPriority.NotRemovable, Nothing)

            End Sub

From what I can see, the problem was being caused by the "CacheItemRemoveCallback" part of the cache.insert line of the code.  I still don't actually know why this is exactly, but I do know that it was that part causing the error.  More information is available here (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconNotifyingApplicationsWhenItemIsDeletedFromCache.asp) if you look at this page you can see that I'm not handling the call back parameter in the right way at the moment, but I was just looking for a quick fix by including the "Nothing" that you can see in the line above.  The errors are certainly gone.

I've also changed the way the code is called so that the refresh function can handle both of the excel datasets (Through the use of the "strObjectName" variable).  I'm still checking for the items in cache on page load of the page using the data - The reason for this is that right now I have no other way of handling the re-insert of the cache items when the dependency item changes.

So the solution isn't quite there yet, but I do get to be rid of the error messages whilst I search for the proper way to handle the cache item being removed.
0
 
PsychotextAuthor Commented:
Think I've got it now:

      'New item in Global.asax.vb

            Shared Sub RemovedCallback(ByVal key As String, ByVal value As Object, ByVal reason As CacheItemRemovedReason)
                  Dim file As New System.IO.StreamWriter("c:\debugtext\cache.txt", True)
                  file.WriteLine(Date.Now.ToString & " - Cache Removed (" & reason.ToString & ")")
                  file.Close()
            End Sub
            
      'Change to RefreshCache in Global.asax.vb
      
            Shared Sub RefreshCache(ByVal strObjectName As String)
                  Dim onRemove As CacheItemRemovedCallback = New CacheItemRemovedCallback(AddressOf RemovedCallback)

                  System.Web.HttpContext.Current.Cache.Insert(strObjectName, GetExcelData(strObjectName), _
                        New CacheDependency(System.Configuration.ConfigurationSettings.AppSettings("dependencyFile").ToString()), _
                        Cache.NoAbsoluteExpiration, Cache.NoSlidingExpiration, CacheItemPriority.NotRemovable, onRemove)
            End Sub

By running the system with the text file outputs on again I get:

      08/12/2004 17:38:12 - Application Start
      08/12/2004 17:38:12 - dsMainExcelData
            RefreshCache run for the main excel data (Called by app start)
            
      08/12/2004 17:38:13 - dsDropExcelData
            RefreshCache run for the dropdown excel data (Called by app start)
      
      08/12/2004 17:38:54 - Cache Removed (DependencyChanged)
      08/12/2004 17:38:54 - Cache Removed (DependencyChanged)
            1st & 2nd cache items removed as I had made changes to the file the cache is dependent on
      
      08/12/2004 17:39:48 - dsMainExcelData
      08/12/2004 17:39:48 - dsDropExcelData
            Page Load event calls RefreshCache for both cache items that are empty
      
      08/12/2004 18:02:01 - Cache Removed (Removed)
      08/12/2004 18:02:01 - Cache Removed (Removed)
            Application is ending, so items are removed from Cache
            
      08/12/2004 18:02:01 - Application End

I believe that as long as I refill the cache for all "CacheItemRemovedReason" barring "Removed", the system should work exactly as expected.  I will probably leave the Page_Load checks in just in case, but I'll monitor their usage and take them out if they do not get called within the first month of running.

Thanks for your help. I think I just needed someone to prod me in the right direction (and to pursuade me to re-write that painful code!). Still lots to learn, but I'm on the right track. :)
0
 
der_jthCommented:
Interesting. Thanks for all the information, I believe I learned something as well.
0
 
PsychotextAuthor Commented:
Apologies... I was jumping the gun.  Whilst the code above does work, it once again fails when using the following code:

    Shared Sub RemovedCallback(ByVal key As String, ByVal value As Object, ByVal reason As CacheItemRemovedReason)
        If Not reason.ToString = "Removed" Then
            RefreshCache(key.ToString)
        End If
    End Sub

With the following error:

      System.NullReferenceException: Object reference not set to an instance of an object.
            at ProgramName.Global.RefreshCache(String strObjectName) in Y:\Websites\ProgramName\Global.asax.vb:line 65
            at ProgramName.Global.RemovedCallback(String key, Object value, CacheItemRemovedReason reason) in Y:\Websites\ProgramName\Global.asax.vb:line 45
            at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)

I know that key / strObjectName isn't null as I'm testing it on the way in.  I've got a nasty feeling that it's got something to do with "CacheItemRemovedCallback" again.  I'm tempted to leave it.... but then I'm not giving up yet!  Trying to find some info on the web, but this doesn't seem to be a particularly well covered topic.
0
 
PsychotextAuthor Commented:
Just as an update.  It's not the "CacheItemRemovedCallback", it appears to be in the GetExcelData function.  Strangely, I'm getting the following message:

System.Data.OleDb.OleDbException: The Microsoft Jet database engine cannot open the file ''.  It is already opened exclusively by another user, or you need permission to view its data.
   at System.Data.OleDb.OleDbConnection.ProcessResults(Int32 hr)
   at System.Data.OleDb.OleDbConnection.InitializeProvider()
   at System.Data.OleDb.OleDbConnection.Open()
   at ProgramName.Global.GetExcelData(String strObjectName) in Y:\Websites\ProgramName\cache.vb:line 99
   at ProgramName.Global.RefreshCache(String strObjectName, String strMode) in Y:\Websites\ProgramName\cache.vb:line 20
   at ProgramName.Global.RemovedCallback(String key, Object value, CacheItemRemovedReason reason) in Y:\Websites\ProgramName\cache.vb:line 151
   at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)

This happens twice (Once for each cache related call).  The strangest thing of all is that it doesn't happen if I choose not to get the data in the refresh cache, but get it from the page_load event instead.  I also have no idea why "cannot open the file '' ", there's no reason for that filename to be empty as the filename is being populated properly.  It's almost as if the function is being called twice at the same time, but I know that's not the case as I'm outputting to a file every time it is called.
0
 
PsychotextAuthor Commented:
Just as a final update to this... looks like the dependency file is being locked by .net when the CacheItemRemovedCallback routine is being run.  Immediately after the routine is finished, the locks are released and the file is available again. I think I'm going to have to find someone a far better than me to fix this one! :)
0
 
der_jthCommented:
Interesting... Are you sure you're properly closing all the connections in GetExcelData? (wondering could it be locked from the previous reading) AFAIK, .net Cache uses a FileSystemWatcher based method for monitoring the file dependency changes; thus it shouldn't certainly lock the file because of the dependency (I don't think .net even opens it).
0
 
PsychotextAuthor Commented:
I really believe that I am, simply as the locking only seems to occur when the cacheItemRemovedCallback is running. I can call this function successfully at any point barring within that loop. Here's the code that gets the data:

    Public Function GetExcelData(ByVal strObjectName As String) As DataTable
        Dim myConnection As OleDbConnection
        Dim strServerPath, strConnection As String
        Dim dtExcelData As DataTable

        strServerPath = "C:\cachetest\file.xls"

        Dim myOleDBCommand As OleDbCommand
        strConnection = "Provider=Microsoft.Jet.OLEDB.4.0; Data Source=" & strServerPath.ToString & ";Extended Properties=Excel 8.0;"
        myConnection = New OleDbConnection(strConnection)

        If strObjectName = "dsMainExcelData" Then
            myOleDBCommand = New OleDbCommand("SELECT [Product],[Part No],[DESCRIPTION],[Q'TY],[TCO],[UK COST],[Date IN] FROM [Sheet1$] WHERE [Product] <> '';", myConnection)
        ElseIf strObjectName = "dsDropExcelData" Then
            myOleDBCommand = New OleDbCommand("SELECT [Product] FROM [Sheet1$] WHERE [Product] <> '' Group By [Product] Order By [Product];", myConnection)
        End If

        Try
            myConnection.Open()
            dtExcelData = ConvertDataReaderToDataTable(myOleDBCommand.ExecuteReader(CommandBehavior.CloseConnection))
           
            'Not needed, but I've added it to make sure.
                  myConnection.Close()

            Return dtExcelData

        Finally
            If Not myConnection Is Nothing Then
                If myConnection.State = ConnectionState.Open Then
                    myConnection.Close()
                End If
               
                'Overkill... but I'm trying everything to make sure the connection is gone
                myConnection.Dispose()
                myConnection = Nothing
               
                OleDbConnection.ReleaseObjectPool()
                myOleDBCommand = Nothing
            End If
        End Try
    End Function

I'm going to try the msdn forums (http://msdn.microsoft.com/newsgroups/default.aspx?dg=microsoft.public.dotnet.framework.aspnet.caching) and if I don't get any luck there I don't know what to do!.
0
 
PsychotextAuthor Commented:
Another thing... it has to grab the dropdown data immediately after the main data, meaning that if the code was locking the file and leaving that lock in place; I wouldn't be able to grab the dropdown data either.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 9
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now