Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty

On a headless (=without console) network server, the 11g JDBC driver used for (java) application connect may cause trouble. In my case, it refused to connect to the DB without any error, trace or log entry. It simply hung. After several hours, it connected one time, and freezed again. Remote debugging done by the development clarified that it locks after calling SeedGenerator() and SecureRandom().

Reason: The JDBC 11g needs about 40 bytes of secure random numbers, gathered from /dev/random, to encrypt its connect string.
But public-available “man 4 random” says:

When read, the /dev/random device will only return random bytes within the estimated number of bits of noise in the entropy pool. /dev/random should be suitable for uses that need very high quality randomness such as one-time pad or key generation. When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.

So far so good, now the question arises: Why does this mystic “entropy pool” runs out of gas?
The answer is as simple as unsatisfying: because too less entropy “noise” was generated by the system. You can check the “filling level” (maybe zero?) of your pool and the overall size of the pool (usually 4096) by issuing

cat /proc/sys/kernel/random/entropy_avail
cat /proc/sys/kernel/random/poolsize

Hint: /dev/random will deliver one new random number as soon as the pool has reached more than 64 entropy units.

So why does my box not generate more entropy noise?
Because only few drivers will fill the entropy pool, first of all keyboard and mouse. Sounds very useful on a server in a datacenter, isn’t it? Some block device and network drivers seem to do so as well, and I have read from guys on the net changing their network card and driver to enjoy this “feature”! But let’s stop ranting, /dev/random is simply made for high security randomness, and if it can’t make sure that randomness is as good as possible in this deterministic world, it stops. Intelligent people have created /dev/urandom for that, like “man 4 random” clearly states:

A read from the /dev/urandom device will not block waiting for more entropy. As a result, if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver. Knowledge of how to do this is not available in the current non-classified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, use /dev/random instead.

Now let’s get back on our JDBC problem. Oracle JDBC 11g seems to use /dev/random by default, which causes usually no trouble on clients running with console access by a user, because his/her unpredictable :) actions will keep the entropy pool well-fed. But to make it usable on a headless server with a latently empty entropy pool, you should do several things, in descending security order (without warranty):

  1. Involve an audio entropy daemon like AED to gather noise from your datacenter with an open microphone, maybe combine it with a webcam noise collector like VED. Other sources are talking about “Cryptographic Randomness from Air Turbulence in Disk devices“. :)
  2. Use the Entropy Gathering Daemon to collect weaker entropy from randomness of userspace programs.
  3. Talk your JDBC into using /dev/urandom instead:
    -Djava.security.egd=file:///dev/urandom

Enjoy Oracle on Linux/Unix!
Yours, Usn

EDIT: Corrected Air Turbulence link

EDIT: Have a look at haveged (collecting good entropy on basis of CPU clock flutter)




You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

37 Responses to “Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty”

  1. Luis Says:

    Oustanding information. I was trying every imaginable combination of params/JDK versions to overcome the problem with Oracle 11g, Liferay EE on Centos 5.4. Until I found your post, and it worked quite fine. Thank you for sharing.

  2. tito Says:

    Very useful!! Congratulations.
    There’s no bound to weirdness one can find in life!!

  3. Toshiba Driver Says:

    Nice posting, very like to read it

  4. hughw Says:

    Thanks for this explanation. /dev/urandom solved my problem.

  5. Munvar Says:

    I am having the same problem, but it didn’t solve my problem. I used -Djava.security.egd=file:///dev/urandom . It din’t work. I am running it on Windows machine. My Java program still failing to connect to my database. Please help.

  6. Mariana Says:

    Even the most weird stuff has a solution ;) It solved my problem :) Thank you!! Very useful!

  7. Somatik Says:

    Thanks a lot! You saved my day. So is there an oracle bug tracker issue related to this?

  8. Timofey Basanov Says:

    Thanks a lot! You have saved my day!

  9. Diagnosing “random” connection resets in 11g | portrix systems Says:

    [...] is also a quite well hidden metalink note on this which also links to this really good blog article. Uncategorized ← Speaking at UKOUG in Birmingham /* [...]

  10. JB Says:

    Per your recommendation, I installed an older fan right above the server next to a micrphone. This is opposed to new Dyson Air Multipliers. We all know that only the wind buffeting caused by the blades will truly engage a random pattern that will allow the entropy daemon to capture the wave patterns properly. By having a bad bearing the noise pattern although repeatable was just random enough to get the entropy daemon garbage cleanup done properly – but occasionally I still lost connections; especially to system 10G. So I then installed a parrot on top of the fan and had it fed my a datacenter troll (DCT – we call him Ed) at random times throughout the day. In effect when the parrot took a poop the random noise of the **** hitting the fan resolved all my connection issues. Including 10G!! You can imagine how happy my manager was to have the end of entrophy issues!! However, every now and then I walk into the data cenetr and he has the parrot on his shoulder pretending to be Captain Morgan. He’s always striking that silly pose with his foot on the barrel. To resolve this I put a honey badger in a parrot suit on top the fan – sure enough the random feedings stopped the random lost connections again. However, one day my manager came in with cuts all over his face. I asked him what the heck happened? He said he was making sure the **** hit the fan, but when it (the ****) looked different he was concerned for Polly (a fore mentioned parrot). When I told him it was really a honey badger we all had a good laugh, and to date we wonder at how when the **** hits the fan things work properly. Frankly when things hang we still ask “Is the **** hitting the fan?” Around here – most of the time it is. Thanks again for the help. Damn it Ed – what the hell are you doing go feed Polly. Geeze. I need to outsource him – slacker.

    COMMENT FROM USN:
    No clue who this clown is. If it’s spam, its really worth reading it. :)
    Usn

  11. Andrew Says:

    Great post. Ran into this issue and have no idea how long it would have taken to figure this out had I not found this blog entry.

  12. Philipp Says:

    Agree with Andrew! Strange Problem, nice Blog Post! Thanks a lot!

  13. Keeping your Maven build fast – BEKK Open Says:

    [...] makes database connection pooling faster for Oracle databases on Linux/Unix systems (see here for [...]

  14. Rick-Rainer Ludwig Says:

    Thank you very much for your blog entry! I was puzzling around for the last week to find out why unit tests broke by time out. An excessive logging and log file reading brought me to network connection and later to the database connection and JDBC aka. ojdbc5. With that information at hand I came to your blog and I got the solution. Thanks, again! Awesome information! It worked for me, too.

  15. gdq Says:

    Had a similar poblem when moving a tomcat from one server to an other; when tomcat started deploying the apps on the new server it hanged with a last log line:

    FINE: Force random number initialization starting

    First solved it by specifying a session handler and random class in context.xml (used default values), but with the option mentioned above, could leave out the context.xml

    Thanks for the info!

  16. Viktor Says:

    Great post! Had the same problem when starting more Instances paralled on a WebObjects Monitor /dev/urandom solved my problem!

    Thank you!!!

  17. Mahesh Says:

    What is difference between file:/dev/urandom and file:///dev/urandom.
    I was using file:/dev/urandom but was still facing problem. The later looks like solving my problem till now, but not sure yet. Have to test more.

    Thanks.

  18. usn Says:

    Hi, I think the only difference is the wrong syntax. :)
    file:///dev/urandom is just the way how to specify a file here.

  19. Raja Says:

    Thanks a lot Usn.
    It clearly saved atleast a week of debugging..

  20. bb Says:

    A very many thanks – worked for us as well. Sometimes its scary just how surreal technology can get..

  21. prem kashyap Says:

    Well I have mentioned the line -Djava.security.egd=file:///dev/urandom in the setEnv.sh of my Apache tomcat server.

    lets see how it goes. I am hoping for the best reading so many success reviews. Just wanted to know any security issues related to the above line!

    Regards
    Prem Kashyap

  22. usn Says:

    Hope you checked the success as described, before enabling it in production. There’s indeed a security impact – your randomness is weaker. That means, more predictable for outsiders. If you are in special danger for encryption security, please consider using another good entropy source as described in the post, like a microphonein the datacenter.

    Regards
    Usn

  23. Satish Says:

    I get below error randomly even after configuring
    -Djava.security.egd=file:///dev/urandom this as java arguments . We are using jboss7.0.2 with oracle 11.

    java.sql.SQLRecoverableException: Closed Connection
    at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.translate(SQLExceptionSubclassTranslator.java:82)
    at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.translate(SQLErrorCodeSQLExceptionTranslator.java:237)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:604)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:638)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:667)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:683)
    at com.mportal.sport.api.subscriber.impl.SubscriberDAOImpl.getSubscriber(SubscriberDAOImpl.java:190)
    at com.mportal.sport.api.subscriber.impl.DefaultSubscriberAdapter.getSubscriber(DefaultSubscriberAdapter.java:130)
    at com.mportal.sport.api.subscriber.impl.SubscriberManager.getSubscriber(SubscriberManager.java:255)
    … 50 more
    Caused by: java.sql.SQLRecoverableException: Closed Connection
    at oracle.jdbc.driver.OracleStatement.ensureOpen(OracleStatement.java:4051)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3563)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
    at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:111)
    at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
    at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:645)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:588)
    … 56 more

  24. This is so obscure, I needed to add it here « Cyclops' Blog Says:

    [...] copied wholesale from Usn’s IT Blog [...]

  25. Sebastian Says:

    Thank you!!!! I spent hours trying to fix this very issue, and your post helped me solve it.

  26. Jeroen Says:

    Thank you!!! I was totally lost on why I couldn’t connect to my database. Thanks to your post everything is working now.

  27. Rakesh Says:

    Thank you very much for this information. Saved me tons of time trying to debug this.

  28. Usn’s IT Blog » Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty | Webscalability Says:

    [...] Usn’s IT Blog » Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty. [...]

  29. Lukasz Says:

    Thanks a lot for this tip! I had spent a week trying to figure out what was causing connection resets before I came across this blog.

  30. David Says:

    Fascinating!

  31. Hari Subramanian Says:

    What an information!! Even after years its helping..

    Thank you So much Martine, you made my day.

    Thanks

  32. Sandeep Says:

    I have Jboss 5.1 application server on windows.
    Oracle 11.2 g on Linux.

    So I have to add -Djava.security.egd=file:///dev/urandom

    on windows client applciaiton server or DB server?

    Can you please answer this.

  33. usn Says:

    Have a look at haveged (collecting good entropy on basis of CPU clock flutter)

    http://www.issihosts.com/haveged/

    Regards
    Martin Klier

  34. Juraj Michalak Says:

    I had the same problem in the past, but this time I was not able to fix it because I saw somewhere that it’s possible to change random number generator globally in jre/lib/security/java.security. There is following line:
    securerandom.source=file:/dev/urandom
    So I thought that it’s already set, but oracle jdbc client was still getting connection reset due to ORA-3136 (timeout, and according to java exception stack trace I saw it’s in authentication).

    After I changed it to:
    securerandom.source=file:///dev/urandom
    (added ‘//’) it works correctly.

    I found explanation here:
    http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom/169338#169338

    /dev/urandom doesn’t really mean ‘/dev/random’, it’s some java’s implementation called NativePRNG. So it’s nicely confusing.
    Maybe the best in order to achieve strong security is to set /dev/random in connection with some additional daemon which feeds /dev/random with entropy (rng-tools and similar…)

  35. dipanjan Says:

    If the driver is not able to read data from the random file and its blocking , then how is that its happening always at the same point , when the java code is trying to execute same query in the database using jdbc driver. can this be justified somehow based on this details..

  36. PCFreak Says:

    haveged helps a lot even without Oracle. It generally builds better entropy. I use it on all my Linux boxes.

  37. simple java program connecting to database server using JDBC has execution time varying all over the place | Yanda Answers Says:

    […] http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-… […]

Leave a Reply