How Is the Infinity Platform Extended and Customized

Application Software Engineers, Custom Solution Developers, and IT Administrators use the same mechanism to extend and build applications on the Infinity platform. Infinity has an XML-based metadata catalog system that uses XML schema definitions to define various features/parts of an application. A catalog on the Infinity platform contains one or more related "specs" (catalog items). A catalog item is an XML document based on a schema that describes a particular type of feature available on the Infinity platform. Examples of a few features types are data forms, data lists, pages, and record operations. The schemas wrap up most of the standards and tenets of the Infinity platform. The Infinity platform allows developers to load XML catalog items/specs into the platform. After they are loaded into the system, these specs are typically referred to as "features." The XML specs must conform to schemas defined in the platform or they cannot be loaded. Since the platform knows what elements to expect in any particular spec type when it is loaded, the platform knows how consume and render that spec type.

What is a Catalog?

A catalog is an assembly that contains embedded catalog items as resources. Custom catalog assemblies built by third-party developers will deploy to the bin\custom folder on the application server. During run-time, the Infinity platform scans the deploy\bin directory to determine the list of available catalog items for the application. As a result, an administrator of the application may copy additional custom catalog assemblies after install to make catalog items available to the application and allow for great flexibility in customizing an application on the Infinityplatform. The Infinityplatform also looks for individual catalog items (XML documents) in the deploy\bin directory in addition to the catalog assemblies.

Even though a catalog item is available as a result of deployment, Infinity will not create the features and entities associated with the catalog item until it is loaded into the database.

How Are Specs Loaded into the Catalog?

Infinity provides various ways to load a catalog item including the CatalogBrowserLoadCatalogItem web service method, a command line utility, and the load spec service revision. All of the ways of loading catalog items invoke the same CLR stored procedure (USP_LOADSPEC) to process the item and create the various objects in the database.

When a spec/catalog item is loaded, where does it go?

When a catalog item is loaded, a row is added to the appropriate catalog item table that contains the XML definition of that spec. For example, it you load a TableSpec, a row is added to the TABLECATALOG table. If you load a QueryViewSpec a row is added to the QUERYVIEWCATALOG. This row contains a good deal of information about the spec as well as the entire spec XML. It is often useful to examine this data when troubleshooting issues.

The LOADSPECLOG table stores every spec loaded into the system. When you load a spec, the LOADSPECLOG is checked to see if the item was loaded previously. If so, it compares the spec with the previously loaded version. If the spec was not altered since it was last loaded, it will skip reloading the spec.

Unless explicitly mentioned below, all specs can be loaded multiple times. Developers can load a spec, then change the XML and reload in order to update the version in the database. This provides nearly unlimited extensibility.

The reason that we put the metadata into the database is that we treat our specs as first-class data entities in the system and we sometimes generate specs on the fly from within the application. For example, when you grant access to a data list to a system role there is a row added to a table called dbo.SYSTEMROLEPERM_DATALIST that has two foreign key columns, one pointing to dbo.SYSTEMROLE and one pointing to dbo.DATALISTCATALOG. Some features within the application also generate specs at runtime. For example, you can create a user-defined data list using the Ad-hoc query tool. When you do that, a data list spec is generated at runtime and stored in the database just like any other data list spec. And some features such as Page Designer let you edit spec definitions. All of those things would be more difficult to implement if the spec content changes had to be written back to the web server disk because usually the web server bits are configured as a read-only disk resource, not as something that we write customer specific application state to. (Another way to put it – sometimes I say that we try to avoid ever "tattooing" the web server with any customer specific state – the idea is that all the web servers can be gold clones of each other with consistent configuration including disk content). Also, in our cloud-based Blackbaud Infinity solutions, just one set of bits/assemblies is shared by multiple application instances, so customer-specific metadata needs to be located with customer-specific data. So while reading from the web server disk would be more efficient than hitting the database (assuming you could create an efficient indexing scheme to look up a spec off disk by ID), it would not be practical to write specs to the web server disk at runtime (actually multiple disks in a web farm scenario), so we have opted to always put metadata in the database given all of the scenarios that need to be supported, and we use caching to mitigate that tradeoff.

Can I Browse the Features That Are Already Loaded?

Absolutely. The articles Application Features, Feature Metadata, and Use "Go To" to View Metadata for a Feature help you explore the various types of features in the system.

When Infinity Needs the Metadata for a Spec, Does It Query the Database for That Metadata or Does It Grab the Metadata from the XML Document That is Embedded in an Assembly on the Web Server?

The answer is that at runtime, the metadata always comes from the database. While the best practice for third-party developers is to embed the custom XML specs into catalog assemblies that get deployed to the web server and then load the spec into the database via the catalog browser, the fact is that at runtime the system is only going to look in the database when it needs the spec metadata. The Blackbaud Product Development teams actually no longer embed the XML files in our assemblies in the latest versions of Blackbaud CRM (2.9 2011 Q3 or later). This is why if you go to the Catalog Browser page in Administration\Application in the shell in the most recent release, you will not see much in any Blackbaud.AppFx.*.dll assemblies.

There were two reasons that the .xml files were ever embedded in assemblies originally as part of the features you get "out of the box." First, we use Visual Studio, and at some point a given set of specs will probably need at least one actual .NET CLR class, so we do end up producing assemblies that go in the web server \bin folder. It is convenient to organize the XML specs that refer to those CLR classes in the same .vbproj project, so from a source code organization standpoint we included those specs in the various projects. Secondly, Blackbaud Product Development (PD) originally embedded resources in the assemblies for the purposes of having a single file with all of the specs in it that could be used for deployment. It makes it convenient to just copy the assembly to the web server \bin folder (which it would need to be there for those CLR classes) and then we could build tooling such as the Catalog Browser to support “loading” the spec from the on-disk assembly into the database. That technique still works and is fine and a good mechanism for custom specs and assemblies. However, a few years ago we changed how we deployed specs that are in-the-box produced by PD developers. Today all specs are always loaded as part of the database revisions that run when you install an update or new version, so by the time the update is complete, all of the specs from Blackbaud PD will have been loaded into the database. Until recently, we continued to embed the specs in the assemblies even though those embedded resources were never used at runtime, but we changed that to cut down on the disk footprint of the .dll assemblies in the \bin folder. We have More than 20,000 specs, so with that change we managed to cut a few hundred megabytes off the on-disk footprint.

After a Feature's Metadata Is Retrieved from the Database, Is It Cached on the Web Server for Performance Reasons?

For performance reasons, we cache the metadata in the web server using the ASP.NET cache feature set. When we fetch the metadata for a given spec from the database the first time, it is cached for N minutes, where N is a configurable setting in the Web.config file. If a request comes in for that same spec’s metadata after N minutes, then we do a fast query on just the rowversion value (aka TSLONG) for the catalog table that is indexed by ID. Since catalog metadata is relatively static the check on the rowversion/TSLONG is a very fast one and only if the TSLONG has changed since the metadata was cached will the metadata again be fetched – but even then it is a very fast single-row-accessed-by-indexed-id type of lookup query. See the documentation for the ASP.NETapp setting value MAX_METADATA_CACHE_SECONDS in the http://{server}/{bbappfx root}/util/appsettinginfo.aspx web.config documentation to determine how long the cache is set to on your system and what the default is.