High "system" CPU usage by SimpleTimer2 threads

[Zlatin Balevsky]

Analysis shows that almost half of the CPU usage on Windows is due to SimpleTimer2 threads bumping in each other.

Performance with:
4 SimpleTimer2 threads: ​http://imgur.com/a/esadK#3
2 threads: ​http://imgur.com/a/ivXMX#2

[Zlatin Balevsky]

Windows 7 x64
Oracle 1.7 64 bit

[zzz]

Is your theory that the SimpleTimer2 implementation (ScheduledThreadPoolExecutor? and its queues and ST2.TimedEvent?.run()) are the problem, or that the TimedEvents? that run on ST2 are "bumping" in their timedReached() ?

If the former, I don't have a lot of ideas. Is this limited to Windows?

If the latter, we should try to get to the root cause.

One possible example of the latter is FragmentHandler? where it queues a timer for every received msg at the (OB or IB) EP. A periodic cleaner could be more efficient. There's other cases I'm sure.

We don't have any visibility into ST2 usage. Something that measures run counts, total run time, and max run time, by TimedEvent? superclass (perhaps using StatisticsManager?, perhaps just a HashMap? local to ST2), could aid in the diagnosis.

So atm I view the 4→2 change as more of a pointer toward the problem than a solution.

[Zlatin Balevsky]

SimpleTimer2 implementation. There is a single queue inside the ScheduledThreadPoolExecutor and the worker threads are either:
a) bumping in each other trying to dequeue the next TimedEvent
b) all waking up when there is only one event in the queue, and all but one are going to sleep again

Reducing the number of threads reduces the contention and idle thread activity in both of these situations. I'm observing similar CPU usage reduction in OSX with Java 1.6.

[zzz]

There's been several changes to ST2 since this ticket, some written by the OP.
There's also the changes for #1672 in 0.9.23
And now we have Java 8.
Could OP please retest?