Concurrency problems under heavy load?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Concurrency problems under heavy load?

daniel_skipjaq
This post was updated on .
Hi,

We have been testing the Petclinic (the Spring sample) application for performance tuning and have noticed an error on the Dandelion libraries when the application is heavily loaded. It seems that when the application is heavily loaded and there is concurrent access to a JSP page where the "datatables:table" tag is used, the following exception appears:

SEVERE: Servlet.service() for servlet [petclinic] in context with path [] threw exception [java.lang.ClassCastException: java.lang.String cannot be cast to com.github.dandelion.datatables.core.extension.Extension] with root cause
java.lang.ClassCastException: java.lang.String cannot be cast to com.github.dandelion.datatables.core.extension.Extension
        at com.github.dandelion.datatables.core.extension.ExtensionLoader.loadExtensions(ExtensionLoader.java:90)
        at com.github.dandelion.datatables.core.generator.WebResourceGenerator.generateWebResources(WebResourceGenerator.java:101)
        at com.github.dandelion.datatables.jsp.tag.AbstractTableTag.setupHtmlGeneration(AbstractTableTag.java:232)
        at com.github.dandelion.datatables.jsp.tag.TableTag.doEndTag(TableTag.java:143)
        at org.apache.jsp.WEB_002dINF.jsp.vets.vetList_jsp._jspService(vetList_jsp.java:150)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
        at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:209)
        at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:267)
        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1221)
        at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1005)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:952)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.github.dandelion.datatables.core.web.filter.DatatablesFilter.doFilter(DatatablesFilter.java:75)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.github.dandelion.core.web.DandelionFilter.doFilter(DandelionFilter.java:157)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

 As I said, this only happens sporadically when there are several threads executing requests at the same time on a JSP page with that tag.

Any idea what might be happening?

Version: 0.10.0
Template engine: JSP
App server: Tomcat
Servlet API version: 2.5
Web browser: Chrome/Firefox
Using Spring/Spring MVC: Yes (in Petclinic app)

Many thanks,

Daniel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

Thibault Duchateau
Administrator
Hi again,

This one worries me a bit more. I need to reproduce first.
Would you share your JMeter configuration or is it private?

Regards,
Thibault
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

daniel_skipjaq
Yes, no problem, we are using the JMeter script attached testplan.jmx with a long ramp-up up to 1000 concurrent threads (creating 1 thread per minute) and no-delay between requests.

It fails on the vets.html page for example, where dandelion datatables are used. I'm not sure which version of Petlcinic we are using, but I can get back to you later with that info. What I know is that it uses the version of dandelion tables posted above.

Many thanks,

Daniel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

Thibault Duchateau
Administrator
Hi Daniel,

I'm currently profiling a custom fork of the PetClinic app that uses Dandelion-Datatbles v1.0.1. The first results sound good to me.
I'll try to re-run it with the v0.10.0.

Could you please re-run a load test with this fork on your part?
=> https://github.com/dandelion/dandelion-core-samples/tree/1.0.1/core-jsp-petclinic

Regards,
Thibault
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

Thibault Duchateau
Administrator
My mistake.. :-/

Actually I'm hitting other errors using the v1.0.1.
The memory footprint seems good.
But some exceptions are raised.
Investigating...

Thanks for the heads-up!

Regards,
Thibault.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

daniel_skipjaq
Hi,

Thanks for your response. I'll wait for the results of your investigation then.

Best,

Daniel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

Thibault Duchateau
Administrator
Got it! Some ConcurrentModificationException are raised while processing options.
On the other hand, I confirm that there is no alert regarding the memory footprint.

I had planned to refactor the options processing in order to make it more generic for future components.
Finally I'll work on this problem sooner than expected (Dandelion-Core 1.1.0 and Dandelion-Datatables 1.1.0)

Thanks again for the heads up!

Regards,
Thibault.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

Thibault Duchateau
Administrator
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

Thibault Duchateau
Administrator
In reply to this post by daniel_skipjaq
Hi,

After spending some hours with Romain, we have finally spotted it!
Two causes led to looooots of exceptions, both of them in TableConfigurationFactory. I'll explain here, hoping that it can help anyone who reads this topic.

1) ConcurrentModificationException
Usually raised when trying to remove an element from a collection/map while iterating over it. Not the reason here, but simply because the map can be cleared by a thread while another reads it: https://github.com/dandelion/dandelion-datatables/blob/1.0.1/datatables-core/src/main/java/com/github/dandelion/datatables/core/option/TableConfigurationFactory.java#L121-123

2) Reference to static data (the most stupid one): https://github.com/dandelion/dandelion-datatables/blob/1.0.1/datatables-core/src/main/java/com/github/dandelion/datatables/core/option/TableConfigurationFactory.java#L139
When calling OPTIONS_BY_GROUP_BY_LOCALE.get(locale).get(group), it references the static OPTIONS_BY_GROUP_BY_LOCALE map. So as soon as a thread processes the passed options, the static reference is updated, thus leading to a complete mess.

Much thanks Daniel. And you made me realize that I hadn't performed load tests since the release of the v1.0.0. It's now done and fixed.
I've also commited some JMeter test plans in the major sample apps for future testing.

The backlog has been updated.

Regards,
Thibault
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Concurrency problems under heavy load?

daniel_skipjaq
Thanks Thibault! That was a quick response and efficient fix :)

Best,

Dan
Loading...