One of the issues that had plagued EvalS (an evaluation performance application/portlet) from the beginning was a performance issue. EvalS was the first jsr 286 that we wrote for the Luminis portal. During the first several releases we worked hard to improve the performance by reducing the # of queries and caching whenever possible. In the past, whenever a person would first load the portal page containing EvalS it would take about 5-6 seconds for it to finish loading the page.

This EvalS performance defect affected all users, only after their initial login. This type of performance was not something we were proud of, so over time we worked on improving the code base, and performance of the backend code. A few months ago, we dedicated some resources to finally fix the problem once and for all.

Our initial assumptions were that the EvalS specific code was slow due to it not being optimized for the number of employees and jobs at OSU. This assumption proved to be incorrect once our development environment included enough random data to match the amount of records in production. After a careful analysis of EvalS and the differences between production & development, a small piece of code external to EvalS, but which EvalS relied on was the identified as the culprit.

The problem:
When a person first accesses EvalS, the application needs to figure out the ONID username of the person. It was this piece of code causing the problem and slowing down the application for the person when they first logged into the application. We never expected this piece of code to be a problem, that’s why we didn’t look into it at first.

The Luminis portal doesn’t store the ONID’s username in the User_ table of the portal. Instead it uses a random # and stores it in the “screenName” column. This is the column where the ONID username would usually be stored. We use an sql query to translate between the random Luminis # assigned to each user and their ONID username. One of the joins this query was using didn’t contain the necessary indexes. This was making the query slow.

The solution:
The fix was rather simple once the culprit was identified. The owner of the external query created a new table that we queried instead. This table contained the necessary data along with needed indexes. EvalS now queries this table and the speed has improved drastically.

We should have challenged our assumptions when we were troubleshooting this performance issue, but we have learned some valuable lessons from our mistakes, which will be helpful in the future. In current and future projects, we now test & analyze the performance of the application early during the development stages. Our development environment now includes enough random data to match the amount of data in production and allow for growth.  Moving forward in this way allows us to demystify application behaviors.