The importance of ThreadDump"S"

 


The Problem

Application that becomes unresponsive is always a pain and find what is going wrong is one of the hardest activity that developers, architects and operators have to deal with.


Recently I had the "plaesure" to see all the VM where an application was deployed crashing for unknown reason: the application did not respond anymore and tomcat was not able to serve any request.


Usually if we look just at the machine, we do not have any information about what is wrong: log stop to be written..we just see that CPU is increasing its usage (and RAM of course).


To really understand what is going wrong, it is important to have a Thread Dump of the JVM to see effectively what each thread is doing at the moment. But there is a problem on that:

if you just take one thread dump you have just a snapshot of the application at a specific time while you are missing the "evolution" of the application


Thread dump overview

For example just looking on the first Thread Dump received, it was not clear what is going wrong. To analyze it, we use one of the best tool which is fastThread.io


The report shows that there are many many TimedWaiting Threads and there are also 2 blocked thread. So what people usually do is to observe the blocked threads and to find what is the problem. This is the principal guilty of the application situation, at the first glance.


Thread Dump mistake

And this is a huge mistake. Thread dumps are ok only if you take more than one and can compare all of them. This is important because you can see the evolution of the application. Blocked doesn't mean that we have a problem, maybe they are using the same resource and in a while the resource will be free to be used, so it's just an instant situation.


Collecting other 2 dumps waiting 2 minutes among them (but you can also have more dumps with less waiting time) you can zip together and upload to the tool and see the comparison:


And if you scroll the comparison and go to CPU spike we see that



So there was 5 threads that were in infinite loop (they are executing the same code, but in different point this is why they are not grouped all together).


The real problem

Infinite loop is not always pure evil, but in this case the loop was related to listing element in a List structure. But at the end of each loop, the list is changed adding one element. The loop was not implemented just using iterator (this prevent list change) but using indexes. So the problem was:


  • Condition on list was always true. The JVM need to check the dimension of the list every loop
  • An element is added at the end of the loop to the list. So the list is increasing the RAM occupied
  • GC is not able to find elements to be discarded because - after million of loops - the list is referring all the elements that can't be removed

The solution was just to use another list, because the goal of the algorithm was to just enrich the list.


Conclusion


So it is important to have a comparison among different Dumps because just analyzing one dump, you can just see only temporary problems that can be a normal situation of a JVM. Remember to use FastThread.io for this purpose because it is free for simple activities, but if you pay you can have ML and experts that can help you to find solution on unresponsive applications













Commenti

Posta un commento

Post popolari in questo blog

HTML and AI - A simple request, and a beautiful response for ElasticSearch without Kibana

Immagine
THE PROBEM I recently installed an ElasticSearch in a Docker environment and I want to be sure that it is working, before waiting the consumers start to connect to it. The Container is started using a volume which was cloned from another Docker installation, so it was not empty: we already have some indexes and some documents. I want to be sure that everything was available but I don't have any Kibana installation (and I can't install it at all). So only API was available and I need to check something  ElasticSearch REST APIs THE FIRST SOLUTION If you ask to Microsoft Copilot how to interact with ElasticSearch api, it will reply by suggesting: curl scripting postman But the interaction is not so human readable and it is quite noisy to write the commands. I want to have something to be easily used but I don't want to spend to much time on that. I need an accelerator THE SECOND SOLUTION I know that there is such an api to ask to ElasticSearch the list of indexes. And I also k...
Continua a leggere

A simple CD using AWS Pipeline

Immagine
  CD is a friend in software development: there is no other way to speed up feedback from your application without having a solid Continuous Deployment. I am skipping continuous integration because all of this examples are related to single repo.. where the only developer allowed to commit code it's me: the integration is with my self :) In this example we will create an automation related to a simple situation where we have a LambdaFunction exposed through an ApiGateway, where the lambda function is running a docker image. There are infinite ways of creating and managing Lambda and pipelines of course, but in this example we will use: CodeCommit to store the code ECR to store the code CloudFormation to create and update all the infrastructure. CodeBuild to create the artifacts we need Let's see it CODE COMMIT     To easily have a repository to store the code and to have a natural integration with AWS echosystem, I decided to use Code commit. Code is structured as other e...
Continua a leggere

Websocket Chat with Lambda (but all in Java)

Immagine
  I don't know if I am not able to perform search on Internet (I doubt it) but every examples of websocket chat created with Lambda was all created using Node.js. And I can cheat on Java, that's my love. So I decided to transform the usual chat into a Java project, by of course performing some changes in order to be easy to check if it works.  THE IDEA The idea behind the project is simple: a simple WebSocket API exposed with api gateway with 4 different route and a unique Lambda who is in charge to serve each request. The change I made is about the name registration: instead of just propagate the message to other connected users, I want to have the way to register a name for chat users so every time a message is published, the receiver can see who sent the message. As I said in other project, this is to be intended for testing purpose, so there are really infinite ways to do have the same behavior maybe in a better way. THE PROJECT The entire project is based on cloud formati...
Continua a leggere