Calling the GetValidReferencingEntitiesRequest method using MetadataService

Hi GaryRasmussen,

I have not checked that method alone but I have worked with Metadata service extensively, I am sure the other entities you are getting are system entities, there must be a filter you will be able to apply on them.

CRM does lot of hiding to present a much simpler dataset when it comes to System related attributes.

Regards,
Chinmay.

And yes as long as the delay is concerned, my answer is "Cache".

Can you please explain what you mean by cache?  Thanks!

Sure. Just like any other ASP.Net application, CRM is also Cache-enabled. I do not know exact technical implementation but they DO lot of client side and server side caching to improve performance.

Here is how you can build your own caching mechanism
http://www.stunnware.com/crm2/topic.aspx?id=metadatacache
http://www.stunnware.com/crm2/topic.aspx?id=metadatacache2

Also you can refer to CRM SDK topic title : Detect Metadata Changes

Basically rather than querying CRM everytime you need some information from the meatdata service, you will cache it [mostly on server sdie] and then query that cache till it expires. It makes a lot of sense to Cache Metadata as once your CRM is up and running frequency of changes in Metadata(Thus expiring the cache) will be reduced by a great magnitude.

Regards,
Chinmay.

Hmm, looks interesting.  My code that returns CRM entity information is a .NET assembly and then other applications will instantiate it and call the public methods.  At what point do I build the cache and where at?  IOW, the 2 are not really connected.

And what is the cache?  I mean maybe I should just add a method that does all the work and creates XML files that the assembly wil read rather than having to query CRM all of the time?

Also, this is not a web application even thought CRM is.  How do I use cacheing?

Yes Caching is amazingly interesting and kind of performance enhancements it can deliver can really surprise sometimes. For the definition of cache kindly refer to: http://en.wikipedia.org/wiki/Cache
Basically you use Cache whenever you know the original data source is slow [relatively] and you need to consume data in a better manner.

Let me take a simple example. There are lots of things to consider when it comes to caching so take this example as an analogy only.

In our computers, we have Hard Disk and we have RAM... now Hard Disk is our main data source and is capable to store large amount of data without any issue. However our processors are REALLY fast when it comes to processing that data. Now if your processor keeps waiting for data to come from your HDD it will be really really painful 'cause no matter how fast your processor is, it cannot process data faster than the rate at what your HDD is providing it. So they introduced RAM... RAM sits between your HDD and processor. It holds a (relatively small) subset of data, data that is currently needed to run O/S and other applications which you are using. If they need more data from HDD, they remove some data from RAM that is not being used and it goes on and on.

Now in our case, Processor is your application, CRM is the HDD and RAM is going to be the cache. Also I just read your comment that this is a WinForms application. I suggest you do it this way, first time you click the button to fetch the entities and it takes 16 seconds right? Alright. Press the button second time and let me know how it goes. I think it should take less than a second but let's see.

Regards,
Chinmay.

I should not have asked what is cache.  I know what cache is.  What I mean is where is the cache.  Say I grab a list of all accounts.  Where do I save this list so that the accounts are cached and my assemply can now read from cache rather than from CRM.

Again, this is NOT a web application.

Yes. I DO understand that this is not a web application and it will be simpler for us.

Kindly read my answer did you try to  load the regarding entities second time? It is very important that you do try that step. See, you access CRM via web service and it's a pretty heavy one too. So when you hit CRM for the first time, your application will have ot instantiate CRM web service and that is a huge operation. but once it is initialized, subsequent calls are going to be really fast.

The cache in your case, will be a List<entity> variable in the assembly itself or better the calling application itself. What you will have to do is to determine how many Accounts[records] you want to keep in the cache? what is the frequency when you will refresh it? etc.

About
"Where do I save this list so that the accounts are cached and my assemply can now read from cache rather than from CRM."
You will not save the records unless and until you want your application to work even if CRM server is not available. You will keep the records in memory. If you want you can use any of the datastore, XML a local access or SQL database but then there are lot of things to consider.

Also what is the volume of data you are expecting? I would strongly recommend to make an in-memory cache and consume it rather than going for a persistent datastore like XML or a local database.

Regards,
Chinmay.

OK, I am with you.  However, I don't think I can do it that way.  An application runs that "might" use my assembly and call some helper functions.  These functions return entity information to the caller as an XML string.

So the primary application runs and calls a function in my assembly.  The first time it runs, it could connect to CRM, cache all the entity information needed (long process), and then return whatever.  But the next day when the application runs, there is no cache.  I can only cache the information after the assembly is used but as soon as the primary application closes, the hook into the assembly is destroyed and the cache goes away right?

Is there a way to persisit the cache even after the application closes other than writing out the data to XML flat files?

Absolutely Yes. You can come up with a webservice that can act as a cache or a Windows Service that stays on the user's system and will cache the data as per the usage. However, could you tell me what exactly you are trying to achieve here?

Regards,
Chinmay.

Ugh, I would hate to write and have to include a service just for caching.

We have a small program (think of it as a module) that is written in an old language and is the only language the main program supports.  We want to add the ability to send email using CRM.  Since this older language supports COM, I wrote a small assembly that can consume the web services necessary send an email using CRM.  Then the old program can now send email using COM and calling functions in my assembly.

Since CRM uses guids rather than email addresses for sending email, I find it also necessary to provide lists of things like Contacts, Accounts, Queues, etc so that the module can populate drop down lists so that the user can select the contact that they want to send to rather than having to know the guid of the contact.

It takes 6 seconds to connect to the web service and another 15 seconds to return the list.  Since there are many drop down lists, you can see the time increases.  15 seconds per list (too long IMO).

So if caching is the only solution, this post is more about how to persist data.  Is there a way to encrypt an XML file?

How do I find out why CRM returns a much shorter list of regarding entity types than GetValidReferencingEntitiesRequest does?  Doesn't CRM use the exact same method?  There is no way to filter the results of GetValidReferencingEntitiesRequest except for the primary entity.

Ofcourse you can encrypt an XML file, it is nothing but a text file with special text[Markup] in it that does not change the content type. So you can encrypt it like any other text file. I would sincerely suggest to populate data on Demand. That is suppose your user wants to send an email, well allow him to search data from CRM I am sure it won't take that long. Are you creating a new CRM service instance for each call? I don't know how you have implemented it but you would want to persist the CRM service instance as long as you can and I think it should increase the speed like anything.

Let me know if you need help with encrypting XML.

Regards,
Chinmay.

No, I connect to the service 1 time, that takes 6 seconds.  Then I might grab an account list, that takes 16 seconds.  A contact list 16 seconds, and so on.  So if you have several pull down lists on your form and you launch the form, it could take 2 minutes to populate the drop down lists.

Perhaps I will just go to a single list and just rebuild the list when the user changes the entity type?

Do you know how do I find out why CRM returns a much shorter list of regarding entity types than GetValidReferencingEntitiesRequest does?  Doesn't CRM use the exact same method?  There is no way to filter the results of GetValidReferencingEntitiesRequest except for the primary entity.

As I mentioned earlier, I will say it again : That logic to filter the entities is listed in CRM Web application i.e. if you are building your own client and then you want use only 4 entities, you will explicitly mention that these are the four entities I want to show and so on.

Also this call explains not existing but *all* entities which are eligibile to have a relationship with email entity. If you want to get the list of entities which in reality do have the relation you will have to retrieve Email entity and then iterate it's OneToManyRelationship as shown below and filter out relationships which are not valid for Advanced Find and then subtract OneToManyRelationships from ManyToOneRelationShips and you will get the list of entities you are expecting.

  // Create the request
                RetrieveEntityRequest entityRequest = new RetrieveEntityRequest();
                // Retrieve only the currently published changes, ignoring the changes that have
                // not been published.
                entityRequest.RetrieveAsIfPublished = true;
                entityRequest.LogicalName = EntityName.email.ToString();
                entityRequest.EntityItems = EntityItems.IncludeRelationships;

                RetrieveEntityResponse entityResponse = (RetrieveEntityResponse)CrmServiceProvider.CrmMetadataServiceInstance.Execute(entityRequest);


                for (int i = 0; i < entityResponse.EntityMetadata.ManyToOneRelationships.Length; i++)
                {
                    if (entityResponse.EntityMetadata.ManyToOneRelationships[i].IsValidForAdvancedFind.Value)
                    {
                        Console.WriteLine(entityResponse.EntityMetadata.ManyToOneRelationships[i].ReferencedEntity + "\t" + entityResponse.EntityMetadata.ManyToOneRelationships[i].ReferencingEntity);
                    }
                }
                Console.WriteLine("-----------");

                for (int i = 0; i < entityResponse.EntityMetadata.OneToManyRelationships.Length; i++)
                {
                    if (entityResponse.EntityMetadata.OneToManyRelationships[i].IsValidForAdvancedFind.Value)
                    {
                        Console.WriteLine(entityResponse.EntityMetadata.OneToManyRelationships[i].ReferencingEntity + "\t" + entityResponse.EntityMetadata.OneToManyRelationships[i].ReferencedEntity);
                    }
                }

Select allOpen in new window

Kindly consider only ManyToOneRelationships, you do not need to filter OneToManyRelationships. I am looking at some things will get back to you soon.

Thank you for the code example!

But I thought that the whole purpose of using the GetValidReferencingEntitiesRequest method was to get back a list of all the entities that an email could be regarding so that you did not need to do all that checking.  It just seems odd to me.

Well it is a heck of a lot closer!  Here is what was returned:  Attached is a screen shot of what CRM is returning.

account
asyncoperation
campaignactivity
contact
contract
incident
invoice
lead
opportunity
quote
rsi_download
rsi_exercisetracking
rsi_functionalcontact
rsi_onlineorder
rsi_productlicense
rsi_productlicenseregistrationkey
rsi_promotions
rsi_renewal
rsi_renewaldetail
rsi_softwarebug
rsi_softwarebuild
rsi_softwareenhancement
rsi_trip
rsi_userevent
rsi_usereventitemdetail
rsi_usereventitems
salesorder
 

Is there a way to get more information about the entity other than just the name "while" looping the response?  Like display name?  I don't see many properties for the ReferencedEntity.

No. to get Display name you will have to call CRM again, let's don't go in that direction. What you can do is you can create an XML and map this schema names and display names there. That will be really quick and will not involve a lot of effort.

BTW, thank you for all your help on this.  You are definately getting the points regardless if we get this to match perfectly or not.  I have a question that is hard to articulate so I am not getting anywhere researching it.

Most all entities have a primary attribute name called "name".  However not all do.  For instance, when looking up contacts, the attribute you want to see is actually fullname. This is known as the primary attribute name but that is misleading becaseu if you research how to get the primary name for an entity, everything I try gives me contactid which is the name of the primary attribute not what I want which is fullname.  Do you know how to look up this value?

 

How does that work for custom entities?  Don't I have to look up the map since it will be different on different systems.  It would be cool if all these entity lookups at the very least gave you back the name, display name, and ID.

No problem. I've been working with CRM for so many years... :P never thought how this list gets populated. Mostly we use CRM as UI and for custom applications we always know what entities we want so things have been really easy. :) this one has been fun. :D

Now coming to the point, primary attribute is not that name, it is always the primary key and it's name will always be in this format: <<entity schema name>>id.

Regards,
Chinmay.

But everything I try to get back "fullname" from a contact entity returns "contactid".  In CRM, if I go to customize the contact entity, there is a primary attribute tab and if you click it, it tells you that the name is fullname.  Even though I know that the actually primary attribute or unique identifier is contactid.

Do you happen to know if there is a way to use contact as an argument and query the metadata service to return fullname rather than contactid?

Tell me how you are querying CRM? I need to see your code to query CRM.

Actually while trying to recreate the code for you I got it to work.

Any idea why my entity list does not include case but the CRM list does?  I definately want to be able to set an email regarding a case.

Incident = case

Gawd, I knew that :(

Thanks!

Here is a completed code example for anyone following this thread in the future.  It returns an xml string containing the name and display name of all entities that a CRM email can be regarding.  The elements are sorted by display name.

public string GetRegardingEntities(string entityType)
        {
            string entities = "";

            try
            {
                RetrieveEntityRequest request = new RetrieveEntityRequest();
                request.RetrieveAsIfPublished = true;
                request.LogicalName = entityType;
                request.EntityItems = EntityItems.IncludeRelationships;
                RetrieveEntityResponse response = (RetrieveEntityResponse)metadataSvc.Execute(request);

                //Add the entity names to an array list so they can be sorted
                ArrayList names = new ArrayList();

                for (int i = 0; i < response.EntityMetadata.ManyToOneRelationships.Length; i++)
                {
                    if (response.EntityMetadata.ManyToOneRelationships[i].IsValidForAdvancedFind.Value)
                    {
                        if (response.EntityMetadata.ManyToOneRelationships[i].SecurityType == SecurityTypes.Append)
                        {
                            names.Add(response.EntityMetadata.ManyToOneRelationships[i].ReferencedEntity);
                        }
                    }
                }
                names.Sort();

                StringBuilder xml = new StringBuilder();
                xml.Append("<Entities>");

                foreach (string name in names)
                {
                    xml.Append("<Entity>");
                    xml.Append("<Display>" + GetEntityDisplayName(name) + "</Display>");
                    xml.Append("<Value>" + name + "</Value>");
                    xml.Append("</Entity>");
                }

                xml.Append("</Entities>");
                entities = xml.ToString();
            }

            catch (SoapException ex)
            {
                error = ex.Detail.InnerText;
            }

            catch (Exception ex)
            {
                error = ex.Message;
            }

            return entities;
        }

private string GetEntityDisplayName(string entityName)
        {
            string displayName = "";

            try
            {   
                RetrieveEntityRequest request = new RetrieveEntityRequest();
                request.RetrieveAsIfPublished = true;
                request.LogicalName = entityName;
                request.EntityItems = EntityItems.EntityOnly;

                RetrieveEntityResponse response = (RetrieveEntityResponse)metadataSvc.Execute(request);
                EntityMetadata metadata = response.EntityMetadata;
                displayName = metadata.DisplayName.LocLabels[0].Label;
            }

            catch (SoapException ex)
            {
                error = ex.Detail.InnerText;
            }

            catch (Exception ex)
            {
                error = ex.Message;
            }

            return displayName;
        }

Select allOpen in new window