Monday, November 10, 2014

How to Tell Programmatically if OpenAM has Finished Starting Up

Update: this post and its thread eventually led to the correct and OpenAM specific mechanism. The short answer is: com.sun.identity.setup.AMSetupServlet (located in openam's core jar) has a public static method isCurrentConfigurationValid() that returns a boolean value. During startup it repeatedly returns false until OpenAM is ready for business and then returns true. Calling core classes after that point solves the problems that I originally outlined in this post. (2014.12.16)


I'm breaking a promise to flesh our more detail from my last post but do plan on coming back to provide those details later. I just couldn't wait to capture and share what I just found. As noted in my previous post I'm adding code to make OpenAM be a RADIUS Authentication server. I've spent some time getting configuration pieces into OpenAM's console. (Post a comment if those notes would be useful in a post.) Now I need to read them at startup time. Simple. Get a handle on the ServiceConfigManager class and go get what you need. Well, not quite.

It turns out that if you call some of these core pieces of OpenAM before they are all brought up you end up shooting OpenAM in the brain. It won't ever recover and no requests to the server will ever get an answer. OpenAM's docs says that the best way to know if OpenAM is up and running is to look at isAlive.jsp rooted at the web-app-root/isAlive.jsp. And it is true. If you request that URL as soon as you start tomcat the browser will spin for about 20 seconds on my box and finally the page will come up indicating Server is ALIVE.

So I thought, there must be some blocking code in isAlive.jsp that does that magic. So I tried using its few lines of code. No go. They too access sensitive ground and OpenAM plays dead. So I stepped back to calling it over http. Now, I can get the webapp's root path from ServletContext. But what port is my server running on? That is only available to servlet code when an incoming http request appears. Until then there is no standard way to get it. PLEASE correct me if I'm wrong there. I'd love to learn that secret. But the specs don't cover it except in the case of an http request.

Then I thought, hey, I can get a Dispatcher and call it without going down to the http protocol. Right? And that is where the magic just happened to appear. Or rather, it disappeared; into null.

In the following code the astute will note that I'm using SpringFramework constructs. Yes I have Spring's Dispatcher servlet defined in OpenAM's web.xml for dependency injection and some REST endpoints that I've added to my local code base. But what I'm doing here can be done with a simple servlet in its init method where you get the ServletContext from the ServletConfig passed to you. That being the case lets look at the code:

@Controller
public class RuntimeServiceController implements ServletContextAware {

    @Override
    public void setServletContext(ServletContext sc) {
        final String cp = sc.getContextPath();
        final String url = cp + "/isAlive.jsp";

        RequestDispatcher d = sc.getRequestDispatcher(url);
        HttpServletRequest req = createRequest();
        HttpServletResponse res = createResponse();
        try {
            d.forward(req, res);
        } catch (ServletException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        // then evaluate captured content
        if (isServerIsUp(res)) {
            // go look at config stuff here
        }
    };
}

When this code ran I got an NPE when calling d.forward(req, res). It doesn't take rocket science to tell that my dispatcher, d, was null. Hmmm. Weird. Maybe the JSP engine is not up yet. If I keep checking will I eventually get a dispatcher? After all, I know that the resource exists at that URL or will at some point in time. So I wrote another version like so:

@Controller
public class RuntimeServiceController implements ServletContextAware {

    @Override
    public void setServletContext(final ServletContext sc) {
        final String cp = sc.getContextPath();
        final String url = cp + "/isAlive.jsp";

        Runnable r = new Runnable() {

            @Override
            public void run() {
                RequestDispatcher d = null;
                long start = System.currentTimeMillis();

                while (d == null) {
                    d = sc.getRequestDispatcher(url);

                    if (d != null) {
                        // now call isAlive.jsp
                    }
                    try {
                        Thread.sleep(100);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                        return;
                    }
                }
            }
        };
        Thread t = new Thread(r);
        t.setName("RuntimeServiceController isAlive.jsp Accessor");
        t.setDaemon(true);
        t.start();
    }

}

I'm launching a Thread to keep testing for the dispatcher so that I don't lock up the web container's start up processes. And this still ended up giving an error. But not until I attempted to call the dispatcher's forward. The dispatcher did indeed eventually appear after about 20 seconds on my box. And that was about how long it took for isAlive.jsp to appear in the browser. Hmmm. Was there a connection? Is OpenAM ready for business once the dispatcher is available?

I stripped out my code for calling isAlive.jsp and added config calls like the code in isAlive.jsp itself and guess what? It worked!!! Now to be fair, I don't know if that is guaranteed to always work. But so far it hasn't failed. Perhaps others familiar with OpenAM can answer that. Any Forgerock gurus around that can answer that? And what about other containers? I don't know. I'm running on tomcat8 on JDK 1.6. If you test it on your server post a comment here and let the rest of us know.

Enjoy.


8 comments:

  1. Spoke too soon. I've bounce tomcat a dozen times since adding this change and just now it failed. Once I got the dispatcher back and called config classes I got the exception below which kills OpenAM. So although this is promising we still have work to do:

    11-Nov-2014 10:25:36.906 SEVERE [RuntimeServiceController isAlive.jsp Accessor] com.sun.identity.authentication.modules.radius.server.config.RuntimeServiceController$1.run ---> GOT IT! After 17195ms
    Failed to create debug directory
    amSMSLdap:11/11/2014 10:25:46:081 AM MST: Thread[RuntimeServiceController isAlive.jsp Accessor,5,main]
    DebugWriter is null.
    amSMSLdap:11/11/2014 10:25:46:081 AM MST: Thread[RuntimeServiceController isAlive.jsp Accessor,5,main]
    ERROR: SMDataLayer:initLdapPool()-Error initializing connection pool Got LDAPServiceException code=-1
    com.sun.identity.shared.ldap.LDAPException: failed to connect to server ldap://ident-local.lds.org:50389 (91)
    Got LDAPServiceException code=-1
    at com.iplanet.services.ldap.DSConfigMgr.getConnection(DSConfigMgr.java:510)
    at com.iplanet.services.ldap.DSConfigMgr.getPrimaryConnection(DSConfigMgr.java:438)
    at com.iplanet.services.ldap.DSConfigMgr.getNewFailoverConnection(DSConfigMgr.java:399)
    at com.iplanet.services.ldap.DSConfigMgr.getNewConnection(DSConfigMgr.java:312)
    at com.sun.identity.sm.ldap.SMDataLayer.initLdapPool(SMDataLayer.java:235)
    at com.sun.identity.sm.ldap.SMDataLayer.(SMDataLayer.java:95)
    at com.sun.identity.sm.ldap.SMDataLayer.getInstance(SMDataLayer.java:107)
    at com.sun.identity.sm.ldap.SMSLdapObject.initialize(SMSLdapObject.java:187)
    at com.sun.identity.sm.ldap.SMSLdapObject.(SMSLdapObject.java:147)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.Class.newInstance(Class.java:374)
    at com.sun.identity.sm.SMSEntry.initSMSObject(SMSEntry.java:292)
    at com.sun.identity.sm.SMSEntry.initializeClass(SMSEntry.java:220)
    at com.sun.identity.sm.SMSEntry.(SMSEntry.java:210)
    at com.sun.identity.sm.ServiceManager.(ServiceManager.java:81)
    at com.sun.identity.authentication.internal.AuthContext.getOrganizationName(AuthContext.java:822)
    at com.sun.identity.authentication.internal.AuthContext.getSSOToken(AuthContext.java:853)
    at com.sun.identity.security.AdminTokenAction.getSSOToken(AdminTokenAction.java:300)
    at com.sun.identity.security.AdminTokenAction.run(AdminTokenAction.java:204)
    at com.sun.identity.security.AdminTokenAction.run(AdminTokenAction.java:76)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.identity.authentication.modules.radius.server.config.RuntimeServiceController$1.run(RuntimeServiceController.java:62)
    at java.lang.Thread.run(Thread.java:744)

    ReplyDelete
  2. Actually, that may have been due to another instance of OpenAM still running which has the embedded OpenDJ instance. Once I killed all of them and tried again it is fine. I'll keep an eye on this. It may work just fine.

    ReplyDelete
  3. Since OpenAM is a Java EE application, it may be beneficial to be aware of the Java EE initialization flow:
    http://download.oracle.com/otndocs/jcp/servlet-3_1-fr-eval-spec/index.html (section 10.12)
    AMSetupServlet is defined with 5, so you may want to use something higher than 5 in your own servlet definition..

    ReplyDelete
  4. Thanks Peter. I'll follow that advice as I roll the RADIUS server pieces in.

    ReplyDelete
  5. Turns out this approach works on Tomcat 8 which is what I currently have on my development box. When we rolled to the servers which are on Tomcat 7 it gets a dispatcher immediately and proceeds to destroy openAM's chances of starting up successfully.

    Solution? I learned via some custom filters that no request gets to any filter anywhere until tomcat logs its, "Server startup in 26159 ms", on either version of Tomcat. So I created a custom IsActive filter that sets a static variable to true for every request passing through it and has a static method for accessing that value. When I watch for that method returning true before contacting OpenAM's core components OpenAM comes up just fine as does the Radius server functionality.

    But I still think it would be helpful if OpenAM had a format mechanism to answer this question.

    ReplyDelete
  6. Did you implement a custom HttpServlet and include it in web.xml like:

    MyServlet
    com.mycompany.MyServlet
    40


    I have trouble to believe that such a servlet would be unable to acquire an admin token. The only problem you can encounter is that your OpenAM isn't actually configured at the time you want to run the initialization logic, but that can be checked using AMSetupServlet#isCurrentConfigurationValid..

    ReplyDelete
    Replies
    1. Well, that XML was stripped by the blog, so let me manually escape it:
      <servlet>
      <servlet-name>MyServlet</servlet-name>
      <servlet-class>com.mycompany.MyServlet</servlet-class>
      <load-on-startup>40</load-on-startup>
      </servlet>

      Delete
  7. I didn't include a custom servlet. The radius server currently has not need for any interaction with any inbound http requests. So I used a ServletContextListener for its implementation. Hence why I've been finding alternative approaches. But your approach should work if I used a servlet as the launching point and place its startup order beyond AMSetupServlet's as you've noted.

    But none of this would have been necessary if I know about AMSetupServlet's static isCurrentConfigurationValid() method!!!! Awesome!!!! I just tested it in place of the filter which I'll be removing. That is the answer that I've been searching for. You rock! Thanks for sharing.

    ReplyDelete