Skipping an unnecessary step

Say you have a simple BPEL process that gets messages from JMS queue through a JMSAdapter, transforms the message, and then calls a down stream service via SOAP.

For such a simple BPEL process, are there any optimization that can be done for performance?

The answer is yes. And the low hanging fruits can be picked from layer between the inbound JMSAdapter and BPEL process.

The Problem

Just open up the WSDL file of the inbound adapter. You will notice something like the following:

<portType name="Consume_Message_ptt">
<operation name="Consume_Message">
<input message="tns:ExpenseRecord_msg"/>
</operation>
</portType>

Notice there is only an INPUT tag but no OUTPUT

tag? This means that the adapter is sending a one-way message to BPEL.

One-way message is one of the

interaction patterns between the caller and the BPEL process that Oracle BPEL supports.

When a BPEL process is initiated by an adapter, the adapter acts more or less like a regular SOAP caller to the BPEL process, except calls will go through a more efficient native binding protocol instead of SOAP. As such, the interaction between the two COULD BE any of the interaction patterns above. But when you wire this adapter to BPEL in JDeveloper, JDeveloper by default chooses the "one-way message" interaction pattern. Hence the WSDL file above.

By default, incoming one-way messages are saved in the delivery service database table dlv_message (it is invoke_message table in 10G). These requests are later acquired by Oracle BPEL Server worker threads and delivered to the targeted BPEL process. Such two-phase interaction can be illustrated as follows:

Figure 1. Default Setting of One-Way Interaction (oneWayDeliveryPolicy=async)

Separating message sending and message processing, and saving message to the database make sense for a lot of cases. The biggest benefit is that allows the sender of message to "fire and forget" - swiftly send the message and get done with it, even though the actual processing of the message make take longer time. One-way interaction is like email: with one click of the button you send an email, while it may make take the recipient of the email 1 hour to read and act on it.

The other benefit is message reliability: Even if the SOA server is down before processing the message, the sender rest for sure that the message is reliably saved to the database so that it can still be processed once the server is up again.

But the same benefits may not make sense for an inbound JMSAdapter.

First the adapter probably doesn't care about fire and forget. What does matter is usually the overall speed or throughput of send+process, instead of just "send".

Second, the message was already saved to the inbound message queue before being picked up. It does not need to be saved at another place for the sake of reliability.

When the benefits of have separate threads to send and process the message

are less meaningful to adapter, its cost becomes more stand-out: it introduces extra latency by saving the message to database. In short, this interaction pattern pays the cost without enough benefits.

The Improvement

If we combine the two threads, and disable the insert into the database table, load against the database is reduced, latency of the insert will be eliminated, and the extra work the container was doing to manage it is skipped. Skipping this layer may lead to improved response times.

It takes two steps to achieve the improvement and these two steps must be done in tandem.

Improvement Step 1: Disabling saving of one-way message to database

In SOA 10.1.3.x, you can set the deliveryPersistPolicy=off.immediate in the bpel.xml of the BPEL process in question.

<BPELSuitcase>
 <BPELProcess  id="ProcessFulfillmentOrderBillingBRMCommsAddSubProcess"   src="ProcessFulfillmentOrderBillingBRMCommsAddSubProcess.bpel">
 ...
 <configurations>
        ...
        <property  name="deliveryPersistPolicy"  encryption="plaintext">off.immediate</property>
    </configurations>
 </BPELProcess>
</BPELSuitcase>

In SOA 11G, you should set the bpel.config.oneWayDeliveryPolicy=sync in the composite.xml of the composite application.

...
<component name="myBPELServiceComponent">
....
<property name="bpel.config.oneWayDeliveryPolicy">sync</property>
</component>

For details of how the oneWayDeliveryPolicy works, please refer to the 13 Oracle BPEL Process Manager Performance Tuning of the Oracle® Fusion Middleware Performance and Tuning Guide. The 10G equivalent is http://docs.oracle.com/cd/B32110_01/core.1013/b28942/tuning_bpel.htm

For details of how to set this property in 11G, please refer to Appendix C Deployment Descriptor Properties of Oracle® Fusion Middleware Developer's Guide for Oracle SOA Suite. For 10G, please refer to Oracle® Application Server Performance Guide 10g Release 3 (10.1.3.1.0)

Figure 2. Results of Improvement Step 1 (oneWayDeliveryPolicy=sync)

Improvement Step 2: Modifying the threads to increase concurrency

Now you have done step 1 to skip saving the one-way message to database. But just by looking at the picture above you may already notice there is a problem.

By skipping saving one-way message to database table, the adapter threads become the sole control of concurrency. The famous BPEL Invoke threads and engine threads do not apply anymore. This essentially means the application becomes single threaded, and the throughput may drop.

That is is why setting deliveryPersistPolicy=off.immeidate (in 10G) or oneWayDeliveryPolicy=sync (11G) should be done in tandem with increasing the number of adapter threads. Otherwise, you will likely create a drop in throughput. Optimal values of adapter threads should be determined through performance testing.

Figure 3. Results of Improvement Step 2 (oneWayDeliveryPolicy=sync with Adapter Threads)

For JMSAdapter in 11G, you can set the adapter threads in the the properties for tuning the thread counts of various adapters are listed as follows.

<service name="dequeue" ui:wsdlLocation="dequeue.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/pcbpel/adapter/jms/textmessageusingqueues/textmessageusingqueues/dequeue%2F#wsdl.interface(Consume_Message_ptt)"/>
<binding.jca config="dequeue_jms.jca">
<property name="adapter.jms.receive.threads" type="xs:string" many="false">10</property>
</binding.jca">
</service>

For 10G BPEL, you set the JMSAdapter receiver threads at bple.xml of BPEL:

<activationAgents>
<activationAgent className=“…" partnerLink="MsgQueuePL">
   ...     <property name="adapter.jms.receive.threads”>5</property>
</activationAgent>
</activationAgents>

For 10G ESB, you set the JMSAdapter receiver threads at *esbsvc file of OESB:

<service name="ListenForNewEmployees" ….>
...
<endpointProperties>
  <property name="adapter.jms.receive.threads" value="10"/>
</endpointProperties>
</service>

The ways to configure adapter threads vary between adapters and product versions. Please refer to documentation for details. I will have another post soon on this subject.

More Benefits

After setting the oneWayDeliveryPolicy=sync (or deliveryPersistPolicy=off.immediate for 10G) AND configure multiple threads, you achieve shorter latency and higher throughput.

But besides the gain in performance, you can also enjoy the following side benefits:

1. Ease of throttling

When oneWayDeliveryPolicy=async, two layers control the incoming rate of messages into BPEL service engine: the adapter threads AND the BPEL invoke threads (see Figure 1 above). And because Invoke Threads are shared among BPEL processes running on the same JVM, you can't just throttle for one process without affecting another.

Now setting adapter threads along with oneWayDeliveryPolicy=sync, you only need to throttle the incoming load with adapter threads, which only affect the SOA component that the adapter calls. You can throttle per adapter individually.

2. Avoid Back Log Messages Flooding the SOA server

If there is an outage of SOA servers and the adapter stops dequeuing from the inbound message queue, the enterprise application on the other side of the queue may continue to enqueue. That builds up a large back log in the inbound message queue.

By the time the SOA servers are restarted, with oneWayDeliveryPolicy=async, the large back log of messages in the queue may flood in the SOA servers, saturating all the Invoke Threads and leave no Invoke Threads to work for other BPEL processes. In contrast, there is no such worry if you throttle the incoming loads to the BPEL process by setting oneWayDeliveryPolicy=sync along with adapter threads.

3. Ease of Monitoring:

Rather than checking both the inbound message queue AND the dlv_message/invoke_message table at database for the flow rate of messages into BPEL service engine, you have one place to monitor: the inbound message queue.