Improving Code Maintainability with Sonar and Jenkins

Problem Statement

Application code growing at an uncontrollable pace, few experienced guys in the team, challenging deadlines. All these made a heady nigtmarish mix for our application from maintenance standpoint.

Issue:

I was tired of the fact that people didn't seem to realize the importance of writing comments. Files after files, methods after methods were being created with no explanation as why they were needed and what purpose they solved.

Solution: 

This time I decided to at least try and put a stop to this never ending nightmare. The solution in my eyes lied in automating code reviews instead of just manually relying on it (which was exhausting and boring). We were already using Jenkins for building our code. I (with the help of a very talented colleague), added SonarQube feature to it. 

Then I tweaked SonarQube quality rules and made comment density a critical component. Initially I have set comment density at 5%. What that means is for every 100 lines of code atleast 5 lines of comments have to be there. If this criteria is not fulfilled for any of the checked in files on a given day, the build for that day will break, requiring urgent attention from team members.

The beauty of this feature is that even if a single line has been modified in a file (say as part of a defect fix), the person checking in the file would be forced to bring the entire file's comment density at par, thereby improving application readability and maintainability.

Below screenshot explains how to configure comment density in SonarQube.

Go to Dashboards>Quality Profiles> (Your quality profile)
Search for relevant rule by entering 'comment' in Search Box. Activate 'Insufficient comment density' rule and set its Severity.
 

Implement Throttling using Apache Camel

Problem Statement

Our application was getting huge surge of orders in small duration and at certain periods of the day only. To quanity further we were getting around 5000-6000 orders in a span of 30-45 mins, around 3 times in a day. Remaining time order volume was 100-300  orders in an hour. To tackle this load, number of app servers were increased from 4 to 8.

Issue
However the concern here was that this costly infrastructure was idle for most part of the day. Going by above stats, the servers were idle (usage much below capacity) for approx 70-80% of the time.

Solution
To optimize this we were asked to explore the option of Throttling.

Throttling: In software terms throttling is a  process responsible for regulating the rate at which application is processing.

Behaviour of application order processing before implementing throttling: Our application works by exposing a REST webservice. Client systems send order xml to this WS and its put on a JMS queue for processing.

Apache Camel Throttling PoC
I began by segregating above behaviour into two different components/routes.
One camel route listened to REST WS and put requests in a folder (say inbox). Another camel route listened to requests coming into this folder and put in on JMS queue but only after being throttled by Camel Throttling. Camel provides an option to define number of requests that need to be picked in a given time interval thereby making sure that even if order volumes surge, only pre defined amount of requests make it to processing stage. Rest stay in file system waiting to be picked up.

Sample code below:
<camel:route>
  <!-- Reading from REST url -->
  <camel:from uri="<REST WS url>" />
  <to
     uri="file:data/inbox?fileName=${header.OrderNumber}-${header.OrderVersion}.xml"
     pattern="InOut" />
</camel:route>
<camel:route>
  <!-- Route to put message from folder to main queue -->
  <camel:from uri="file:data/inbox" />
  <!--  Using camel provided throttling. Defined no of requests that can be processed in given time period -->
  <throttle timePeriodMillis="60000">       
   <constant>1</constant>
   <camel:log
    message="Sucessfully processing service order ${headers.OrderNumber}-${headers.OrderVersion}.xml" />
   <camel:to uri="file:data/outbox" />
  </throttle>
</camel:route>

Apache Camel Redelivery Policy

Problem Statement

Continuing on the problem described in my previous post I wanted that once an exception occurs in my route, instead of trying to poll the request again, the route should sleep for a while say half an hour. This would give sufficient time for servers to stabalize and avoid unnecessary polling.

Issue
I used camel redeliveryPolicy to do this. Since i wanted that the request be polled until it was successfully process and not for some predefined max retries, I specified only redeliveryDelay.
   <camel:onException>
        <camel:exception>java.lang.Exception</camel:exception>
        <camel:redeliveryPolicy redeliveryDelay="50000" />
        <camel:log message="Default error handler was called"></camel:log>
    </camel:onException>

Even though onException was getting invoked delay was not taking effect.

Solution
On my colleague's advice i added maximumRedeliveries tag as well and finally delay took effect. Code snapshot that worked below:
 <camel:onException>
  <camel:exception>java.lang.Exception</camel:exception>
  <camel:redeliveryPolicy maximumRedeliveries="1" redeliveryDelay="50000" />
  <camel:log message="Default error handler was called"></camel:log>
 </camel:onException>

So seems like in redeliveryPolicy, maximumRedeliveries and redeliveryDelay go hand in hand.

Apache Camel - Read file recursively instead of moving it to .camel folder

Problem Statement

Read a file from input folder, check memory consumption on server and process it only if its below threshold limit else let the request remain in input folder.

Issue

Request was getting moved to .camel folder after camel route read it as a result it was not being picked up recursively for processing.

Solution

In your camel route add a processor that checks memory consumption. If its above limit the processor should throw an exception. This will mark message as failed and file consumer wont put it in .camel folder. Instead it will be picked up for processing again.

Sample Spring DSL below:

<camel:camelContext xmlns="http://camel.apache.org/schema/spring">

<camel:route>

<!-- Route to put message from folder to JMS queue if memory consumption is below limit-->

<camel:from uri="file:data/inbox"/>

<camel:process ref="checkMemoryConsumption"/>

<camel:convertBodyTo type="String" />

<camel:log message="Sucessfully processing service order"/>

<to uri="<jms queue"

pattern="InOut" />

</camel:route>

</camel:camelContext>

If you look above code snippet closely, you will that I am using convertBodyTo tag of Camel. This is being used because when reading request xml from folder and directly putting it on JMS queue I was getting below error:
org.apache.activemq.command.ActiveMQBytesMessage cannot be cast to javax.jms.TextMessage
java.lang.ClassCastException: org.apache.activemq.command.ActiveMQBytesMessage cannot be cast to javax.jms.TextMessage
        at com.qwest.mdw.listener.jms.JmsListener.start_in_thread(JmsListener.java:116)

        at com.qwest.mdw.listener.jms.JmsListener$1.run(JmsListener.java:158)