Exposing subsets of a model for review

For many organizations exposing the model for review via the Ontology Review Tool is a great way to get feedback from Subject Matter Experts, but sometimes we only want to expose specific subsets to specific users.

The Ontology Review Tool exposes information inside the selected Semantic Enhancement Server (SES) index, in a follow on to my recent post that describes how to publish multiple models to separate SES indexes this posts describes how to publish different sections of the same model to different SES indexes.

So, lets take an example, in a recent project we were working with a client using the NCI (National Cancer Institute) Thesaurus, we can see the top level concepts below, they wanted to expose two different hierarchies for review by separate groups – ‘Pharmacologic Substance’ and ‘Disease, Disorder or Finding’

NCI Hierarchy

To separate these out into separate SES indexes, we first need to tell the Publisher to do this, we need to edit the Publisher configuration file.

In the default configuration the publisher is configured to create a single SES index, so first we need to tell it that there are going to be 2 SES indexes created, in the below extract I rename the configuration sections for the indexes to be relevant to the sections in the model

<object id="TaxonomyDBConfig" class="publisher.bean.config.common.TaxonomyDBConfig">
	<property name="rulebaseSetConfigs">
		<list>

			<ref object="SubstanceBaselineCollection"/>
			<ref object="DiseaseBaselineCollection"/>

		</list>
	</property>
...

I can then configure the baseline process itself to only read the correct subsections of the model, firstly I used the existing default ‘BaselineCollection’ definiton and altered it to

  • Match the above naming
  • Set ‘useAllTerms’ to false
  • Set ‘startTerms’ property to my selected start term
  • Altered ‘muscatIndexer’ property to give it a unique name so I could have two
<object id="SubstanceBaselineCollection" class="publisher.bean.config.flow.BaselinerCollection">
	<property name="useAllTerms" value="False"/>
	<property name="startTerms">
		<list>
			<value>Pharmacologic Substance</value>
		</list>
	</property>
	<property name="muscatIndexer" ref="SubstanceMuscat4Indexer"/>
</object>

I can then copy this section and create a second configuration for the ‘Disease’ set of terms

<object id="DiseaseBaselineCollection" class="publisher.bean.config.flow.BaselinerCollection">
	<property name="useAllTerms" value="False"/>
	<property name="startTerms">
		<list>
			<value>Disease, Disorder or Finding</value>
		</list>
	</property>
	<property name="muscatIndexer" ref="DiseaseMuscat4Indexer"/>
</object>

So, now we have defined what terms are going to be used for each of the baseline processes we now need to tell publisher what index to write it to, again I start with the default Muscat4Indexer configuration and this time edit.

  • Name to match above configuration
  • indexName
<object id="SubstanceMuscat4Indexer" class="publisher.bean.outprocessor.Muscat4Indexer.Muscat4Indexer">
	<property name="indexName" value="substance_model"/>
	<property name="maxAssociatedRelationships" value="1000"/>
	<property name="maxPathPerTerm" value="10"/>
	<property name="maxBatchSize" value="50"/>
	<property name="termsPerTransaction" value="1000"/>
	<property name="sesConfigFile" value="${ses_config}"/>
	<property name="notifyURLs">
...

I then copy that and configure the second instance

<object id="DiseaseMuscat4Indexer" class="publisher.bean.outprocessor.Muscat4Indexer.Muscat4Indexer">
	<property name="indexName" value="disease_model"/>
	<property name="maxAssociatedRelationships" value="1000"/>
	<property name="maxPathPerTerm" value="10"/>
	<property name="maxBatchSize" value="50"/>
	<property name="termsPerTransaction" value="1000"/>
	<property name="sesConfigFile" value="${ses_config}"/>
	<property name="notifyURLs">
...

And the final step is to then configure the Semantic Enhancement Server to recognize the two indexes that we specified in these two files, we edit semaphore-ses.xml

We can add the two new index names into the indexes section e.g.

<property name="indexes">
	<list>
		<value>disp_taxonomy</value>
		<value>substance_model</value>
		<value>disease_model</value>
	</list>
</property>

and then add the definition for these new indexes, I repeat the default section and change the id to match the name of the index defined above, and the location to again be unique


<bean id="substance_model" class="index">
	<property name="directory"><value>C:\Program Files (x86)\Smartlogic\Semantic Enhancement Server\data\substance_model</value></property>
</bean>

<bean id="disease_model" class="index">
	<property name="directory"><value>C:\Program Files (x86)\Smartlogic\Semantic Enhancement Server\data\disease_model</value></property>
</bean>

Now, I can publish this model using the new configuration and when finished I can access either of my two SES indexes when creating a review (or from any other application that calls SES):

MultipleIndexes_SESIndexes

Exposing Multiple Models to Downstream Systems

A number of clients are using multiple models and want to expose them either into the Ontology Review Tool or as end points for consumption into different downstream systems. Each of those systems would primarily connect to the Semantic Enhancement Server (SES), so to be able to take two models and make them both available we have to configure both the Publisher and the Semantic Enhancement Server to use separate indexes for each model.

Let’s take the example of a pharmaceutical company, they might use multiple industry standard taxonomies e.g. MeSH, MedDRA, SNOMED, NCI, RxNorm, ICD9, ICD10 etc

If we wanted to publish the two models, we firstly would create two separate publisher configuration files, you can just copy and rename the configuration files e.g.

semaphore-pub-MeSH.xml
semaphore-pub-MedDRA.xml

It is not critical but often helpful to update the description of the configuration file as this helps the users select the correct file when publishing.

<property name="description" value="MeSH publisher configuration"/>

Then in each publisher configuration file we need to tell it to use unique index names, this is done by changing the following section – in the below I created a unique indexName, changing it from the default disp_taxonomy, we would create different names in each of the publisher configuration files

<object id="Muscat4Indexer" class="publisher.bean.outprocessor.Muscat4Indexer.Muscat4Indexer">
		<property name="indexName" value="mesh_model"/>
		<property name="maxAssociatedRelationships" value="1000"/>
		<property name="maxPathPerTerm" value="10"/>
		<property name="maxBatchSize" value="50"/>
		<property name="termsPerTransaction" value="1000"/>
		<property name="sesConfigFile" value="${ses_config}"/>

Then we need to edit the Semantic Enhancement Server configuration file to recognize the two indexes that we specified in these two files, we edit semaphore-ses.xml

We can add the two new index names into the indexes section e.g.

<property name="indexes">
	<list>
		<value>disp_taxonomy</value>
		<value>mesh_model</value>
		<value>meddra_model</value>
	</list>
</property>

and then add the definition for these new indexes, I repeat the default section and change the id to match the name of the index defined above, and the directory to again be unique


<bean id="mesh_model" class="index">
	<property name="directory"><value>C:\Program Files (x86)\Smartlogic\Semantic Enhancement Server\data\mesh_model</value></property>
</bean>

<bean id="meddra_model" class="index">
	<property name="directory"><value>C:\Program Files (x86)\Smartlogic\Semantic Enhancement Server\data\meddra_model</value></property>
</bean>

I can then open my first MeSH model, publish that using the semaphore-pub-MeSH.xml configuration, then open the second MedDRA model and publish it using the semaphore-pub-MedDRA.xml, and two indexes will have been created that can be accessed via Semantic Enhancement Server and therefore Ontology Review Tool and other downstream systems.

Working with Multiple Hierarchical Relationships

Introduction

In Semaphore there is the ability to define multiple hierarchical relationships which is useful for displaying your model in different contexts, for example, different teams that organize the model in different ways.  What may not be clear is that these different hierarchical relationships can drive the user experience throughout the Semaphore experience.

Creating a Model with Multiple Hierarchical Relationships

Within Ontology Manager you can define multiple hierarchical relationships using the “Configure” -> “Structure” menu option on the “Relationship Types” tab.  An example is as follows:

As you associate terms in the model in a hierarchical fashion you will be prompted to indicate which hierarchical relationship you wish to use.

When displaying the model in Ontology Manager, in the “Hierarchical” view you can right click and select the hierarchy relationship you wish to have the system use to display the tree:

Publisher

Publishing a model with multiple hierarchical relationships means this information is available in Semantic Enhancement Server but also any application that uses it.

Semantic Enhancement Server

The “browse” service in Semantic Enhancement Server (or the “hierarchy” service if you are using the REST syntax) by default uses the default hierarchical relationship type for the model.  If you wish to use an alternative hierarchical relationship then for “browse” you can add the parameter “hiertype” but for REST you use the “hierarchies” service.  For example:

http://server/ses?tbdb=disp_taxonomy&service=browse&hiertype=NT2

Or:

http://server/ses/disp_taxonomy/hierarchies/NT2/roots.xml

Showing Different Views of the Model

After the model information is published into Semantic Enhancement Server (SES) then various other applications that use SES to display the model can be configured to show the different hierarchical representations of the model that are present.

Semaphore Workbench

A simple modification of the Semaphore Workbench allows any hierarchical relationship defined in the model to be used to drive the user experience.

If you alter the “<web application>/WEB-INF/jsp/ort/browse.jsp” you can select what hierarchical relationship to use to display the model.  So, with the example model we are working with here adjust the following line:

url: getSESRestUrl() + '/hierarchy/roots.json',

To read:

url: getSESRestUrl() + '/hierarchies/NT2/roots.json',

And the following line:

relation_abbreviation: 'NT',

To read:

relation_abbreviation: 'NT2',

Then the model will be displayed using this relationship both when displaying the top level list of terms but also when using the new “tree” view of the model.

Semaphore for SharePoint 2010

In the Semaphore Service Application in Semaphore for SharePoint 2010 when configuring the model you have the ability to specify the hierarchical relationship name that is used when building the tree both in the term store and in the manual term selection display.

The same option also exists in the new Semaphore for SharePoint 2013 solution.

Semaphore 3.6.1 Release

The Smartlogic team is pleased to announce the availability of Semaphore 3.6.1

This is a point release which brings a number improvements and fixes over Semaphore 3.6, integrates many aspects of the feedback provided by customers who have already deployed 3.6, but also includes our first release of the Semaphore for SharePoint 2013 integration.

Updated components include: Classification Server, Semantic Enhancement Server, Publisher, Ontology Manager, Ontology Server, Semaphore Workbench and the Semaphore for SharePoint 2010 integration.

The release can be downloaded from the customer support portal at https://portal.smartlogic.com/support/downloads and the corresponding documentation – including the release notes – is available from https://portal.smartlogic.com/documentation/3.6.1

What does Content Intelligence do for my organization?

Every vendor with a complex product suite struggles to describe the rich functionality their product suites deliver and how that adds value to the prospective customers.  We were working on exactly this problem and realized it was a multi-dimensional model with many axis – The business benefits and advantage, related to sectors and applications and delivered by components through their functionality.

So we modeled this as an ontology and made the results available in the Ontology Review Tool.

To see what we think our solution provides, please click on the link below.   You will be taken to the start point of the model where you can select one of the top level nodes, or you could search for an aspect, say our capability in “Enhanced Enterprise Search” or the impact for the role “Knowledge and Information Management”.  When the “bubbles” visualization appears, click on any node to move around.

http://workbench.smartlogic.com/SW/view/public/ort/browse/26#/

Using LDAP Authentication with Semaphore Workbench

Semaphore Workbench comes with the ability to authenticate users against a Windows Active Directory domain (if using a Windows machine) or against users hard-coded in the configuration.  There are situations where neither of these authentication mechanisms are appropriate particularly if you already have an authentication mechanism already in place.

The good news is that with a few modifications to the standard Semaphore Workbench distribution you can authenticate your users against LDAP.  This article discusses LDAP but similar processes would be involved for other authentication mechanisms (perhaps a candidate for another blog entry).

In this article am assuming that you have a technical understanding of LDAP and a bit of understanding of web application configuration and deployment.

Upgrading an Existing Installation to use LDAP

If you already have Semaphore Workbench installed and do not want to lose your existing configuration then the simplest way to update the configuration for LDAP is to do the following:

  1. Add an appropriate LDAP user to the list of “Reviewers” on the “Setup” page. This is to ensure that you can login to the application once you have configured it to use LDAP!  Note that any other existing reviewers will have to updated to use their appropriate LDAP user names as well.  See below for what is considered a LDAP “user”.
  2. Install the missing JAR files and update the configuration files.  Follow the installation instructions below but skip the first step (as the solution has already been deployed).

Installing Semaphore Workbench with LDAP Support

The following is a summary of the process:

  1. Deploy Semaphore Workbench (see the “Semaphore Workbench – Installation Guide” for details) but do NOT initialise it, that is, do not perform the initial database configuration (as per the instructions), simply deploy the web application WAR file to the JSP-servlet container.
  2. Stop the web application or JSP-servlet container hosting Semaphore Workbench (for example, Apache Tomcat).
  3. Download the following libraries from http://www.springsource.org/download/community required for Semaphore Workbench to communicate with LDAP: “spring-ldap-1.3.1.RELEASE-minimal.zip” (found under “Spring LDAP”) and “spring-security-3.0.4.RELEASE.zip” (found under “Spring Framework”).
  4. Copy the file “spring-ldap-core-1.3.1.RELEASE.jar” from within the “spring-ldap-1.3.1.RELEASE-minimal.zip” archive into the /WEB-INF/lib folder of the Semaphore Workbench web application.
  5. Copy the file “spring-security-ldap-3.0.4.RELEASE.jar” from within the “spring-security-3.0.4.RELEASE.zip” archive into the /WEB-INF/lib folder of the Semaphore Workbench web application.

This is where it gets interesting as you now have to update the Semaphore Workbench configuration files.  Below find changes in bold to the default configuration files (note that these are extracts and not the full files).  You will need to provide relevant LDAP details for the elements enclosed in “<<>>” and may also need to adjust the parameters to “<sec:ldap-authentication-provider>” to reflect the details of your LDAP installation (for example, how “users” are defined).

  • <web application>/WEB-INF/config/common/waffle-auth.xml
...
    <sec:ldap-server url="ldap://<<ldap-server>>:389/<<dn-name>>" manager-dn="<<manager-dn>>" manager-password="<<password>>"/>

<!-- spring authentication provider -->
    <sec:authentication-manager alias="authenticationProvider">
    <!-- For Window, use the waffle authentication: -->
      <!-- <sec:authentication-provider ref="waffleSpringAuthenticationProvider" /> -->
      <!-- For linux, use the dummy provider below instead: -->
      <!--
    <sec:authentication-provider>
        <sec:user-service>
            <sec:user name="admin" password="admin" authorities="ROLE_USER" />
        </sec:user-service>
    </sec:authentication-provider>
   -->

   <sec:ldap-authentication-provider user-search-filter="(uid={0})"
                                                  user-search-base="ou=users"
                                                  group-search-filter="(uniqueMember={0})"
                                                  group-search-base="ou=groups"
                                                  group-role-attribute="cn"
                                                  role-prefix="ROLE_"/> 
... 
  • <web application>/WEB-INF/config/common/webmvc-context.xml
...
    <!--  fake context with static getBean...handy -->
    <bean id="SpringApplicationContext" lazy-init="false"/>

    <bean id="ldapAuthProvider" class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider" />

    <!--  Spring security stuff -->
    <!-- <bean id="waffleWindowsAuthProvider" class="waffle.windows.auth.impl.WindowsAuthProviderImpl" /> -->

    <bean id="negotiateSecurityFilterProvider" class="waffle.servlet.spi.NegotiateSecurityFilterProvider">
        <constructor-arg ref="ldapWindowsAuthProvider" />
    </bean>

    <bean id="basicSecurityFilterProvider" class="waffle.servlet.spi.BasicSecurityFilterProvider">
        <constructor-arg ref="ldapWindowsAuthProvider" />
    </bean>
...

Once you have made these changes you can then restart the web application.

If this is a new installation when you load the Semaphore Workbench URL into a browser you should see the “Configure Database” screen so you can simply enter your LDAP user account into “Your Login ID” (this should not be the full LDAP DN but simply the user name – where “user name” is defined as per the information you have configured in the “<sec:ldap-authentication-provider>” tag in the configuration, as per the above), adjust the rest of the settings as appropriate then press the “Submit” button.  You can now continue on with configuration of Semaphore Workbench as per the installation guide.

If this is an upgrade of an existing installation you can simply login using the LDAP user you previously added to the list of “Reviewers” for the application.  All previously existing configuration should still be available and function correctly.

Semaphore Workbench LDAP User Configuration

Any LDAP users you want to be able to log into Semaphore Workbench to perform classification reviews (for example) will need to be set up as “Reviewers” within the application on the standard “Setup” page.  When doing this specify only the user name from LDAP rather than the full LDAP DN for their “Login ID”.

Display of additional information in Ontology Review Tool

As more and more people are adopting Ontology Review Tool as the primary way to expand the audience for access/review of their models, we are getting a number of customization requests, mostly to display specific notes (term information) stored in the model on the main Ontology Review Tool view.

Ontology Review Tool already exposes a number of fields on this view, to enable them all you need is to create note fields in Ontology Manager and enter the appropriate values (don’t forget to re-publish the model..).

The current supported notes fields are:

  • Term Description
  • site_title
  • site_description
  • site_URL
  • image_title
  • image_URL

For example this would take the following information in Ontology Manager:

defining note fields for use in Ontology Review Tool

And display is as the following in the Ontology Review Tool:

If you already have data in existing note fields then it is still possible to expose these fields but you do to dig a little deeper (and understand some code). To do this you can edit browse.jsp from the Ontology Review Tool install directory, in my example I had deployed the Ontology Review Tool into Apache Tomcat and therefore my file is located: C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ORT\WEB-INF\jsp\browse.jsp

In this file you need to edit the onNodeSelected function, to create new variables, gather the note information into them and then display them in your preferred manner.

Exposing selected term classes in the visualisation tool

Some use cases require the visualisation tool to only expose selected term classes. It might be the case when it is used as an exploration tool on a portal for instance.

This can be quite simply achieved by configuring the Publisher to only baseline specific term classes into the Semantic Enhancement Server. You will need to open the Publisher XML configuration file, locate the BaselineCollection section and add the following lines to it:

    <property name=”classes” >
        <list>
            <value>Concept</value>
        </list>
    </property>

The next time you publish the resulting SES index will only contain terms from the “Concept” class; of course you may want to tailor this to your actual model classes!