Threads Life Cycle

In Java many times we need to face the state of Pool. (also in Java 1.4 certification exam :P )

Interesting articles :

https://howtodoinjava.com/java/multi-threading/java-thread-life-cycle-and-thread-states/

https://dzone.com/articles/difference-between-blocked-waiting-timed-waiting-e

https://examples.javacodegeeks.com/core-java/threads/java-thread-join-example/

Eclipse MAT++

1. Retained vs Shallow heap

Shallow is the real size of the object in the heap, meanwhile Retained is the complete memory space used by all the references.

https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Fconcepts%2Fshallowretainedheap.html

https://plumbr.io/blog/memory-leaks/how-much-memory-what-is-retained-heap


2. Heap diagram:

Many times, it is not very known the states of objects in Heap memory:

https://mechanical-sympathy.blogspot.com/2013/07/java-garbage-collection-distilled.html


3.  “GC root path”.
Mat can calculate the full path from GC to any class, this way we can see big size objects.

Now, MAT will start calculating the memory graph to show the paths to the GC root where this instance is referenced. This will show up with another page, showing the references as below:

https://dzone.com/articles/java-out-of-memory-heap-analysis


4. MAT resources
A small good tip is about the necessary resources to run mat.

As a rough estimate if the number of objects is N and the number of classes C, it might take at least T bytes to parse and build the dominator tree where:

T ≈ N * 28.25 + C * 1000 + P

P is the space used by the DTFJ or HPROF parsers. For a PHD file, the space could be:

P ≈ C * 1000

Memory Analyzer uses additional memory for caching index files, so performance will be better if there is more memory available than the minimum required to parse a dump.

from: https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Ftasks%2Fconfigure_mat.html

5. command line:
Sometimes a good choice to process several heapdumps is to execute thru command line:

sh ./ParseHeapDump.sh /heapdumps/java_7935_heapdump_4p.hprof org.eclipse.mat.api:suspects org.eclipse.mat.api:overview org.eclipse.mat.api:top_components 

 It will create the following reports:

- org.eclipse.mat.api:suspects
- org.eclipse.mat.api:overview
- org.eclipse.mat.api:top_components

Enterprise Technical Architecture

Enterprise Architecture != Enterprise Technical Architecture 

From O'Reilly.

JMX Remote (visualvm)

Arguments for JVM thus VisualVM can connect remotely JMX :

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

TCP/IP Illustrated

RFC793 has several diagrams, in particular, from book TCP/IP Illustrated, Vol. 1: The Protocols, and concise information for this protocol.

IBM has a good replication:

Table 1. TCP state transition description table

TCP connection state	Abbreviation in MVS™ console	Abbreviation in TSO or UNIX shell	Description
LISTEN	Listen	Listen	Waiting for a connection request from a remote TCP application. This is the state in which you can find the listening socket of a local TCP server.
SYN-SENT	SynSent	SynSent	Waiting for an acknowledgment from the remote endpoint after having sent a connection request. Results after step 1 of the three-way TCP handshake.
SYN-RECEIVED	SynRcvd	SynRcvd	This endpoint has received a connection request and sent an acknowledgment. This endpoint is waiting for final acknowledgment that the other endpoint did receive this endpoint's acknowledgment of the original connection request. Results after step 2 of the three-way TCP handshake.
ESTABLISHED	Estblsh	Establsh	Represents a fully established connection; this is the normal state for the data transfer phase of the connection.
FIN-WAIT-1	FinWt1	FinWait1	Waiting for an acknowledgment of the connection termination request or for a simultaneous connection termination request from the remote TCP. This state is normally of short duration.
FIN-WAIT-2	FinWt2	FinWait2	Waiting for a connection termination request from the remote TCP after this endpoint has sent its connection termination request. This state is normally of short duration, but if the remote socket endpoint does not close its socket shortly after it has received information that this socket endpoint closed the connection, then it might last for some time. Excessive FIN-WAIT-2 states can indicate an error in the coding of the remote application.
CLOSE-WAIT	ClosWt	ClosWait	This endpoint has received a close request from the remote endpoint and this TCP is now waiting for a connection termination request from the local application.
CLOSING	Closing	Closing	Waiting for a connection termination request acknowledgment from the remote TCP. This state is entered when this endpoint receives a close request from the local application, sends a termination request to the remote endpoint, and receives a termination request before it receives the acknowledgment from the remote endpoint.
LAST-ACK	LastAck	LastAck	Waiting for an acknowledgment of the connection termination request previously sent to the remote TCP. This state is entered when this endpoint received a termination request before it sent its termination request.
TIME-WAIT	TimeWt	TimeWait	Waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.
CLOSED	Closed	Closed	Represents no connection state at all.

Also, a sequence diagram, where Wikipedia is a good contribution (as usual):

- open: 3 way
- close: 4 way

http://www.icir.org/christian/sock.html

https://www.inetdaemon.com/tutorials/internet/tcp/3-way_handshake.shtml

http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm

Armchair architecture

Armchair architecture anti-pattern, most common in organizations :(

From O'Reilly.

Oracle Traffic Director tuning

• Number of acceptor threads
  
  Acceptor threads receive client requests and put them in the connection queue. When an Oracle Traffic Director instance starts, it creates the specified number of acceptor threads for each listener. If the number of acceptor threads for a listener is not specified, Oracle Traffic Director creates one acceptor thread per CPU on the host.
  
  Too many idle acceptor threads place an unnecessary burden on the system, while having too few acceptor threads might result in client requests not being accepted. One acceptor thread per CPU, which is the default setting, is an acceptable trade-off in most situations.
  
  For HTTP 1.0 workloads, which necessitate opening and closing a relatively large number of connections, the default number of acceptor threads—1 per listener—would be suboptimal. Consider increasing the number of acceptor threads.
  

https://docs.oracle.com/middleware/1221/otd/admin/perf.htm#OTADG799

Description of ''Figure 14-1 Connection Handling in Oracle Traffic Director ''

Description of ''Figure 14-2 Connection Handling in Oracle Traffic Director with Keep Alive Enabled''

https://medium.com/wlsdmforweblogic/how-to-monitor-oracle-traffic-director-otd-components-and-instances-65e90c0cd3c3

inefable

inefable

 

Del lat. ineffabĭlis 'indecible'.

1. adj. Que no se puede explicar con palabras.

Stuck Threads summary

base_domain -> Environment -> Servers -> Server -> Configuration/Tuning:  Stuck Thread Max Time: 600

Stuck Thread Max Time
The number of seconds that a thread must be continually working before this server considers the thread stuck.
For example, if you set this to 600 seconds, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use.
In Web Logic Server 9.x and later, it is recommended that you use the ServerFailureTriggerMBean in the OverloadProtectionMBean. The ServerFailureTriggerMBean transitions the server to a FAILED state after the specified number of stuck threads are detected. The OverloadProtectionMBean has options to suspend or shutdown a failed server.
MBean Attribute:
ServerMBean.StuckThreadMaxTime

base_domain -> Environment -> Servers -> Server -> Configuration/Overload:  Failure Action

Failure Action
Enable automatic forceshutdown of the server on failed state. The server self-health monitoring detects fatal failures and mark the server as failed. The server can be restarted using NodeManager or a HA agent.

base_domain -> Environment -> Servers -> Server -> Configuration/Overload:  Stuck Thread Count

Stuck Thread Count
The number of stuck threads after which the server is transitioned into FAILED state. There are options in OverloadProtectionMBean to suspend and shutdown a FAILED server. By default, the server continues to run in FAILED state.
If the StuckThreadCount value is set to zero then the server never transitions into FAILED server irrespective of the number of stuck threads.

base_domain -> Environment -> Servers -> Server -> Configuration/Overload:  Max Stuck Thread Time:

Max Stuck Thread Time
The number of seconds that a thread must be continually working before this server diagnoses the thread as being stuck.
For example, if you set this to 600 seconds, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use.
MBean Attribute: ServerFailureTriggerMBean.MaxStuckThreadTime

-->   ServerFailureTriggerMBean.MaxStuckThreadTime >>  ServerMBean.StuckThreadMaxTime


more info:

http://www.munzandmore.com/2012/ora/weblogic-stuck-threads-howto

https://docs.oracle.com/cd/E14571_01/apirefs.1111/e13952/pagehelp/Corecoreserverservermonitorthreadstitle.html

Capacity Planning

from Thinking Architecturally by Nathaniel Schutta