Data enter Readiness Checklist

Owned by Deniz Oğuz
Last updated: Nov 27, 2019
7 min read

Caching

The add-on has 5 main features and there are several caches to make it perform fast in each area.

To check cache sizes in bytes we have used "Retained Size" reported by JVisualVM. Although we have limited size of each cache in terms of items in the cache. Size of each item may change depending on size of used strings, whether a field is null or has a value. Also sometimes a list of objects is stored as a field. Depending on the number of items in the list, cache size may change too. We have also measured only cache entries, cache keys are very small in size mostly primitive value like Long. 

Size of all of the above caches combined should be 6.3 + 26.4 + 12.9 + 25.4 = 71Mbytes. This number is just the size of actual objects we store in the cache. But Atlassian Cache stores lost of extra information for each cache entry and for cache itself. For example last access time, pointers to previous and next entries etc. This will make size of caches larger but not too much.

Bundle & Bundle Content Cache 

Not all projects requires bundle feature, only big projects with lots of independently releasable components needs this feature. Also number of bundle releases in a project is generally much less than number of component releases. Our customers combine all hardware and software component releases in a package and release this package to their customers.

There are 2 main caches:

  • (State #1) Project ID to Bundle Cache: Stores bundles (packages) of projects and the second cache (se below) stores content (component & version pairs) of these bundles. This cache enables us to get all bundles of a project without hitting the database. Cache size is 50. We don't expect every project to use bundles feature. This cache is a remote cache. We invalidate entries of these caches whenever a project component or version is updated instead of replicating. Also they expire after 60 minutes. According to JVisualVM, each BundleImp averages 147 bytes. If each cache project contains 20 bundles we should have a maximum cache size of 50 * 20 * 147bytes = 0.15Mbytes .
  • (State #2) Bundle ID to Content Cache: Stores component version pairs of a bundle. Cache size is 1000 and this is a remote cache with replication strategy set to invalidation. Also they expire after 60 minutes. According to JVisualVM, each BundleContentImp averages 88 bytes. If each bundle contains 10 component versions we should have a maximum cache size of 1000 * 10 * 569bytes = 5.7Mbytes.

There are 3 caches are mostly used to speed up our bundle related custom field indexing and JQL functions (deprecated), and grouping for Structure add-on. All of them are remote caches with maximum size of 1000 entries and expires after 10 minutes. Maximum size of each cache is 1000 * 147bytes = 0.15Mbytes.

  • (State #3) Bundle ID to Bundle Cache: Stores higher level information (name, version, description etc.) for a bundle, not content (component version pairs) of a bundle. We index name of bundle in a custom searcher. Our custom field stored bundle Id, but we index name of the bundle. So during indexing we use this cache to lookup bundle name from bundle id.
  • (State #4) Version ID to Bundle Cache: Sometimes affect/fix version of an issue determines in which bundle it is. This cache is used to speed up this lookup. 
  • (State #5) Component Version to Bundle Cache: Sometimes bundle of an issue is determined by current components and versions of an issue. This cache is used to speed up this lookup.


Combined total size of all of these 5 caches are: 6.3Mbytes.

Component Versions Caches

We have two caches.

  • Components to Component Versions (State 6): Keeps which versions are valid for a given component combination (Jira allows multiple component for an issue). This is the most used cache because it is queried every time an issue is displayed or components of an issue changes. We use a special serializable cache key and we have affected by JRASERVER-65249 so we update thread's class loader temporarily. Cache size is limited with 1000 and they expire after 60 minutes and cache is replicated by invalidation asynchronous. According to JVisualVM each cache entry is 121bytes. If we assume that there are 30 versions for each component we should have a maximum cache size of 1000*30*121bytes = 3.6Mbytes.
  • Project to Component Versions (State 7): Keeps all valid component version pairs of a project and used for component versions management page. Cache size is limited with 100 and they expire after 60 minutes and cache is replicated by invalidation asynchronous. According to JVisualVM each component version information is 568bytes. If we assume there are 400 component versions for each project we should have a maximum cache size of 100 * 400 * 568=22.8Mbytes.

Total size of these two caches are: 26.4Mbytes.

Version Hierarchy Cache (State #8)

Used to speed up version hierarchy construction and reduce database lookup for displaying version hierarchy and JQL operations related with version hierarchy.

  • Project to Version Hierarchy Items Cache: Cache size is limited with 100 and they expire after 60 minutes and replicated via invalidation. Cache key is project id and corresponding object is a list of  serializable object with a few Long and integer fields.

Configuration Caches (State #9)

There are 5 different areas handled by the add-on and not every project need to use all of the functions. Also how a feature works depends on user configurable parameters. For this reason the add-on needs to get some configuration objects before it can do anything on the UI. We request subproject configuration, subcomponent configuration and component version configuration objects on every page load (where an issue can be created/displayed). There are 3 caches here, but each cache stores only one object. Replicated via invalidation. Max size of extra heap is < 10KB (there are at most 3 entries in this cache).

Subcomponent Cache

Subcomponents feature allows management of components as a component tree. 

  • Project to Component Hierarchy Cache (State #10): On issue view/edit/create dialogs we add a subcomponent picker when pressed opens an inline dialog showing subcomponent tree of the project. We want this lookup to be fast. For this reason we need to cache this information for each project. We use a cache of 100 entries. It is a remote cache with invalidation replication. This cache is a remote cache with cache invalidation. According to JVisualVM each ComponentHierarchy item is 638bytes. If we assume 150 items in a subcomponent hierarchy of a project we should have a maximum cache size of 100 * 150 * 638bytes = 9.6Mbytes.
  • Link ID To Partial Component Hierarch Cache (State 11): We allow users to query subcomponents with a "component in subcomponentsOf(....)" JQL. This requires us to locate components recursively under a given component. To speed up calculation of these component we use this cache. Cache has 100 entries. This cache is a remote cache with cache invalidation. We store list of component name and Ids for each entry and size of this object is just 32 bytes. If there are 10 items in each level of hierarchy maximum cache size is 1000 * 10 * 32bytes = 3.2Mbytes.

Total size of these two caches is around 12.9Mbytes.

Subprojects Cache

Subprojects feature allows projects to be organized into a tree hierarchy and adds a picker to select a project from this hierarchy. It also provides "project in subprojectsOf(.......)" JQL.

  • User to Project Hierarchy Cache (State 11): Since every user has different project access permissions this hierarchy differs by user and also whether this dialog is opened for creating and issue or for just displaying the hierarchy. This cache allows us to locate hierarchy for a user very fast. We use a cache of 100 entries. This cache is a remote cache with cache invalidation. According to JVisualVM each ProjectHierarchy item is 507bytes. If we assume that there are 500 projects added to project hierarchy we should have a maximum cache size of 100 * 500 * 507bytes = 25.4Mbytes.


Database: Additional tables

We don't use @NonNull constraint for active objects. We need to verify the active objects record every time we use it anyway. Jira doesn't allow us to create foreign key constraints on components, projects or versions tables. So every time we read a record from database we still need to verify that it is still referencing a valid project or component. Adding a @NonNull constraint on any of these fields will not save us. We listen for component, version and project events and update our own database records accordingly but an administrator may temporarily disable our add-on delete some versions and he may re-enable the add-on. We need to recover from invalid references to components and versions. Also as soon as we add a @NonNull constraint we need to hardcode name of these columns when creating active object records, we don't like this. We don't say that adding @NonNull constraints help, it helps. For example one of our database upgrade task was written to set project_id column to correct values. It wasn't expected to be null but it did. We are just saying that there are downsides of this approach too. 


Database: Schema Upgrade Tasks

Our latest and valid upgrade task belongs to 2016. We had a bug in our add-on and project_id of some records were not set correctly. We have fixed this using an upgrade task. Our other upgrade tasks where used to copy value of a column to a new column. But it turned out to be a bad idea and we have commented out body of those upgrade tasks. Today even in a large instance this task should not affect any record. 


Event Handling

The app uses EvenPublisher.publish method to publish an application specific event, SettingsUpdatedEvent. We use this event to break a circular reference between two components and to invalidate caches of the ad-on when setting is updated. Because there are some settings which affects which versions are valid given a list of components. 

We also listen a lot of Jira specific events: ProjectDeletedEvent, ProjectUpdatedEvent, ProjectComponentDeletedEvent, ProjectComponentUpdatedEvent, and events derived from AbstractVersionEvent. We mostly listen them to invalidate some caches and delete update some database records. We do not perform any long-running actions. They may trigger update/delete of a few database row and invalidation of caches. For example if a project is deleted, we delete corresponding component versions, bundle contents and subcomponent hierarchy. None of these events are too frequent to cause any problem. We don't listen any IssueEvent.  


Transaction Management & Resource Usage

We don't use pagination because we don't think we need it. The add-on mostly operates within the context of a project. It loads subcomponent tree of a project, or bundles of a project, or component versions of a project. Even in a large instance number of components or versions within a project is way below than a few thousands. 

We use following code block to manage transactions:

Transaction tx = Txn.begin();
try {
	....
	tx.commit();
} finally {
	tx.finallyRollbackIfNotCommitted();
}

XSS Prevention

All UI of the app is implemented with React and react escapes string by default. This provides some level of  build-in protection against XSS attacks. But we also used OWASP's HTML Sanitizer library at server side of the app to sanitize user inputs. Our app mostly about relationship between existing components and versions rather than user input.