Too many open files

The following problem may occurr in WLS:

Caused By: java.net.SocketException: Too many open files
  at java.net.Socket.createImpl(Socket.java:447)
  at java.net.Socket.connect(Socket.java:577)
  at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:161)
  at oracle.net.nt.ConnOption.connect(ConnOption.java:159)
  at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:428)
  at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:506)
  at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:595)
  at oracle.net.ns.NSProtocol.connect(NSProtocol.java:230)

This is a common issue when  File Descriptors are not enough for the OS processes.

Checking the most open FD appear WLS Managed Servers and they are almost on the limit:

4076 376 /u04/Middleware/WLS12c/jdk/bin/java-server-Xms1024m-Xmx2048m-XX:PermSize=256m-XX:MaxPermSize=512m-Dweblogic.Name=WLS02

4063 32526 /u04/Middleware/WLS12c/jdk/bin/java-server-Xms1024m-Xmx2048m-XX:PermSize=256m-XX:MaxPermSize=512m-Dweblogic.Name=WLS01

Checking with lsof -p <pid> we see a lot of open jar files from war files, because we have several war files deployed.

Check:

$ ulimit -n
4096

Therefore the solution is to set a bigger soft & hard limit.