Send Message To RabbitMQ in Transaction

What do I meant by sending message in Transaction ?

Sending message in transaction mode mean if the transaction is succeed then message/event should be published. But in case of transaction failure or rollbacks the message/event shouldn't be published.

In many Enterprise Application transaction is the bread and butter of the architecture. Traditional architecture is highly dependent on transaction. Even in new era of Microservices  there comes the need to send the messages across the components in some what transaction way to maintain the data integrity across the component.

Let's take some simple example of microservice: 

Here there is simple use case once the order is placed it will notify the user about the order confirmation. In microservice architecture, order and notification are two different component having their own JVM and DB. Definitely this flow should be async and also consistent. 

Here there is a need/requirement  that notification should not trigger if the order placement fails. Notification should only be delivered if the order transaction succeed. So there is need to send message to RabbitMQ  ( Any Broker) in transaction mode.

Currently RabbitMQ  doesn't participate/support the transaction there is way for two phase commit, but there are pitfalls which I am not going to discuss here. 

So question comes how we can leverage the RabbitMQ power to use it under transaction answer is Orchestration. 

Based on my experience there are following options each having their pos and cons.

Option -1 Orchestration Using RabbitMQ Itself. 

For this option create some broker framework which can get embedded into the application or it can run standalone.  

1. Here application create the payload and generate unique hash value.
2. Under transaction it send message to temp queue and writes the hash value to some table in DB.
3. Broker Framework listen to the temp queue and upon receive of any message it checks for the hash value stored in step-2.
4. If the hash value is present it forward the message to original queue and if not present it pushes back to the temp queue.
5. After no of retry the message will get discarded / moved to error queue reason being transaction might not succeed and hash value is never written to DB. 

Challenges:

• If the transaction takes longer the broker will receive the message before the transaction ends and due to which retry might get triggered more frequently.
• If volume is high as in many case message queued back after retry will take longer time to reach destination.

+Ve:

• Highly scalable. In case of embedded it autoscale as the node scale.
• Easy to monitor and maintain.
• Event Driven

-Ve:

• Data Cleanup
• More delay in case of longer transaction time

Option -2 Orchestration Using Poller.

This is very traditional approach where poller will poll the transaction data and publish the same to queue.

1. App stores the payload data in transaction. ( DB Isolation Level read committed).
2. Poller polls the data and push it to queue.
3. Once the data is push the data is cleared from DB via update / delete.

Challenges:

• Poller scaling.
• Heavy read and write operation in DB.
• Poller memory issues (How much data to poll ).

+Ve:

• Truly transaction.

-Ve:

• Difficult to scale.
• High Maintenance and tuning requirements. 

Option -3 Orchestration In Memory.

Here the message will be stored in memory and will delivered to MQ after transaction succeed.

1. Application will store the message in memory till the transaction completes.
2. Once the transaction is completes in memory message is pushed to queue.
3. Shutdown hook is implemented which will write the in memory data to disk if shutdown is triggered so that once the application comes up it will able to check the transaction and push to queue.

Challenges:

• Transaction complete triggering.
• Memory management to some extent.
• Shutdown and Restart.

+Ve:

• Very easy to implement ( Remember to take concurrency effect).
• Scalable.

-Ve:

• Rollback cleanups
• High maintenance in multi node scenario.

Option -4 Orchestration via coordinator.

Here the message will be send to rest broker which can act like coordinator which will coordinate for the transaction processing.

1. Application submit the payload to the broker which return the unique token back in response.
2. On post processing of the transaction the application submit the token back to the broker, and broker send the message to queue.
3. Broker work on time based it will hold the token for stipulated time and will discard the message if the token is not received in stipulated time.
4. Optionally if broker needs to scale then broker might store the message temporarily in some distributed cache like redis etc.

Challenges:

• Transaction completing triggering. ( What if the node goes down after the transaction is complete and was not able to submit the token).
• Complex development and monitoring.

+Ve:

• Scalable

-Ve:

• High Monitoring and fallback.

Conclusion:

Which option to choose depends on the requirement.  Example Option-3 is suitable for low volumes and option-1 and option-4 can be used for high volume loads. Option-2 which is pure transactional stuff, but not event driven and difficult to scale needs rethinking before adopting it.

If you believe in KISS (Keep it simple Stupid) principle then go for Option-1. As per me this is easiest to implement, fault tolerance and highly scalable

Before starting, evaluate various options via POC and find out the best which suits you. 

Happy coding.

JUNIT - Method Contract Testing

What is method contract testing:

Form the developer stand point it is very important to write Junit. I my career I had seen many different method/process/ways people write junit. Some write serious test cases, some writes for just sake of writing the test case and some test cases even puzzled me what exactly test case want to accomplish. 

Now as everyone is moving towards continuous integration (CI), automated builds, self-testing and rapid development writing good junit become the most critical part of development cycle.

In any development the most important aspect to test is Object, Method and its behavior with respect to change in its/dependent object state and evaluation path along with non functional aspects.

Any good JUNIT should revolve along the first two, change in state and evaluation path where as non functional aspects should be covered more precisely in integration testing.

While doing JUNIT developer needs to concentrate to method under test and all input / dependencies needs to be mocked. Test exactly how method will react to input, change in object state. Doing so if each and every method has good test case then eventually you will end up in writing good test case which can cover almost all scenarios ( That depends upon how good test case is written).

Coming to the point each method under test has certain method contracts. Following are different types of method contract:

1) Input Contract.

2) Output Contract.

3) Class Contract.

3) Exception Contract.

4) Behavioral Contract.

5) Non functional Contract.

Input Contract:

1) Null check - Method may or may not check null validity for input parameters.

In this case either current method or calling method should have null check unit test case.

2) Input Modification - Method may or may not modified the input objects.

In this case proper assert needs to be there for input object passed to check for modifications.

Output Contract:

1) return value contract - Based on the signature method can return new object or existing one.

Here proper asserts needs to be there to check for new object or existing object modifications.

2) return by reference - Method can return by modifying the input object which is passed by reference.

Proper assert needs to be there to unit test the same.

3) void contract - Method doesn't return any value.

Assert all the input / class object for any modification.

Class Contract:

1) Class contract - Method can change / initialize the accessible class /static variables.

Provide proper assert to track this changes.

Exception Contract:

1) forwarded exception - Method forward the exception instead of catching.

Mock the exception and assert the same.

2) new exception - Method creates and throws new exception.

Assert the new exception whether it is thrown or not.

Behavioral Contract:

Method can have one or many behavioral contract.

For all behavioral contract proper unit test and assert needs to be there.

Example:

If null is passed as input I will return null object which caller needs to handle.

If dependent call throw exception I will not rollback the changed input /class object.

Non functional Contract:

Method can have various non functional aspect like logging , transaction, concurrency etc.

All non functional aspect should be cover as part of integration testing and it should not be included as part of unit testing.

Heap Dump Basics

Taking and using the heap dump information  is very critical for successful career in Java Development. It will be very hard to believe that if you had done successful Java development, you had never taken or analyzed heap dump.  In my career  I had done many production deployments and  solved many critical production and non production issues where I think without heap dump it will be very tough and some time almost next to impossible to solve certain issues. I had be approached by many how to take and analyse heap dump especially who are new or not mature to Java development. So this blog might help them to get some basic to some detail information about the heap dump in step by step order. Lets start with exactly what is heap dump ?.

What is Heap Dump:

Heap dump is a snapshot of the memory of a Java process at a certain point of time. It contains the information about objects loaded in memory, classes,

garbage collection roots, thread stack, local variables.

Using this information any body can get the following details:

1) How many classes loaded.

2) How many objects created.

3) How many heap space available and occupied.

4) What were the JVM parameters/options used for that run.

5) Heap used by each object type.

6) Thread status.

7) Thread call stack - exactly which method was getting evoked during that time.

8) Unreachable objects and many more. But this information is at the point of heap dump and it doesn't reflect the JVM past state.

When to take Heap Dump and why :

1) When your application seems hanging.
2) When there is sudden spike in memory.
3) When application crash due to out of memory issue.
4) When you want introspection into JVM at particular time or event.

Triggering of Heap Dump:

1) Triggering Heap Dump under live running application.
2) Triggering Heap Dump when the application goes out of memory.

Note: When you trigger the heap dump all the active threads are stopped (duration depends on the heap size) . This can lead to performance issue as many critical application cannot tolerate such activity.

Ways to Trigger Heap Dump:

Heap Dump on out of memory: 

java  -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=   ....

or add to JAVA_OPTS if you are using tomcat etc.

Manually taking heap dump of running application:

Find the process id of the application:

jps -lv or ps aux | grep java.  -- Recommended is jps 

then take heap dump with jmap
jmap -dump:file=<> <>
jmap -dump:file=heap_dump.hprof 4208

Manually taking heap dump using the UI tool:

Start jvisualvm --> Navigate to the process --> go to monitor tab --> Click Heap Dump Button.

Knowing the object size and no of instances:

jmap -histo:live <>
jmap -histo:live 4208


Note you can also take the heap dump by enabling the JMX.

In next blog I will give the tips to analyze the Heap Dump