VisiBroker for Java Developer’s Guide : Managing threads and connections

Managing threads and connections
This section discusses the use of multiple threads in client programs and object implementations, and will help you understand the VisiBroker thread and connection model.
Using threads
A thread, or a single sequential flow of control within a process, is also called a lightweight process that reduces overhead by sharing fundamental parts with other threads. Threads are lightweight so that there can be many of them present within a process.
Using multiple threads provides concurrency within an application and improves performance. Applications can be structured efficiently with threads servicing several independent computations simultaneously. For example, a database system may have many user interactions in progress while at the same time performing several file and network operations.
Although it is possible to write the software as one thread of control moving asynchronously from request to request, the code may be simplified by writing each request as a separate sequence, and letting the underlying system handle the synchronous interleaving of the different operations.
Multiple threads are useful when:
Thread and connection management occurs within the scope of an entity known as a server engine. Several default server engines are created automatically by VisiBroker, which include thread pool engines for IIOP, for LIOP, and so forth. Additional server engines can be used and created in a VisiBroker server by applications. See the example in <install_dir>/examples/vbroker/poa/server_engine_policy/Server.java.
Server engines are created, configured, and used independently. The creation and configuration of one server engine does not affect other server engines in the same server. Usually, each server engine has one transport end point, called the listen point/socket.
The relationship between server engines and POAs is many-to-many. Each server engine can be used by multiple POAs, and each POA may also use multiple server engines.
Server engines can consist of multiple Server Connection Managers (SCMs). An SCM is composed of managers, listeners, and dispatchers. The properties of managers, listeners and dispatchers can be configured to determine how the SCM functions. These properties are discussed in “Setting connection management properties”.
Listener thread, dispatcher thread, and worker threads
Each server engine has a listener and a dispatcher thread. The listener thread is responsible for:
The dispatcher determines which threads to send requests.
Each server engine uses a certain number of worker threads to receive and process requests. Different requests may handled by different worker threads. For a given request, the request reading, processing (include server side interceptor intercepting), and replying are all handled by the same thread. The number of worker threads used by a server engine depends on:
Thread policies
The two major thread models supported by VisiBroker are the thread pool (also known as thread-per-request, or TPool) and thread-per-session (also known as thread-per-connection, or TSession). Single-thread and main-thread models are not discussed in this document. The thread pool and thread-per-session models differ in these fundamental ways:
The default thread policy is the thread pool. For information about setting thread-per-session or changing properties for the thread pool model, see “Setting dispatch policies and properties”.
Thread pool policy
When your server uses the thread pool policy, it defines the maximum number of threads that can be allocated to handle client requests. A worker thread is assigned for each client request, but only for the duration of that particular request. When a request is completed, the worker thread that was assigned to that request is placed into a pool of available threads so that it may be reassigned to process future requests from any of the clients.
Using this model, threads are allocated based on the amount of request traffic to the server object. This means that a highly active client that makes many requests to the server at the same time will be serviced by multiple threads, ensuring that the requests are quickly executed, while less active clients can share a single thread, and still have their requests immediately serviced. Additionally, the overhead associated with the creation and destruction of worker threads is reduced, because threads are reused rather than destroyed, and can be assigned to multiple new connections.
VisiBroker conserves system resources by dynamically allocating the number of threads in the thread pool based on the number of concurrent client requests by default. If the client becomes very active, new threads are allocated to meet its needs. If threads remain inactive, VisiBroker releases them, only keeping enough threads to meet current client demand. This enables the optimal number of threads to be active in the server at all times.
The size of the thread pool grows based upon server activity and is fully configurable, either before or during execution, to meet the needs of specific distributed systems. With the thread pool model, you can configure the following:
Each time a client request is received, an attempt is made to assign a thread from the thread pool to process the request. If this is the first client request and the pool is empty, a thread will be created. Likewise, if all threads are busy, a new thread will be created to service the request.
A server can define a maximum number of threads that can be allocated to handle client requests. If there are no threads available in the pool and the maximum number of threads have already been created, the request will block until a thread currently in use has been released back into the pool.
Thread pool is the default thread policy. You do not have to set up anything to define this environment. If you want to set properties for the thread pool, see “Setting dispatch policies and properties”.
Figure 8
The figure above shows the object implementation using the thread pool policy. As the name implies, there is an available pool of worker threads in this policy.
Figure 9
In the above figure, Client application #1 establishes a connection to the Object Implementation and a thread is created to handle requests. In the thread pool, there is one connection per client and one thread per connection. When a request comes in, a worker thread receives the request; that worker thread is no longer in the pool.
A worker thread is removed from the thread pool and is always listening for requests. When a request comes in, that worker thread reads in the request and dispatches the request to the appropriate object implementation. Prior to dispatching the request, the worker thread wakes up one other worker thread which then listens for the next request.
Figure 10
As the above figure shows, when Client application #2 establishes its own connection and sends a request, a second worker thread is created. Worker thread #3 is now listening for incoming requests.
Figure 11
The above figure shows that when a second request comes in from Client application #1, it uses worker thread #4. Worker thread #5 is spawned to listen for new requests. If more requests came in from Client application #1, more threads would be assigned to handle them, each spawned after the listening thread receives a request. As worker threads complete their tasks, they are returned to the pool and become available to handle requests from any client.
Thread-per-session policy
With the thread-per-session (TSession) policy, threading is driven by connections between the client and server processes. When your server selects the thread-per-session policy, a new thread is allocated each time a new client connects to a server. A thread is assigned to handle all the requests received from a particular client. Because of this, thread-per-session is also referred to as thread-per-connection. When the client disconnects from the server, the thread is destroyed. You may limit the maximum number of threads that can be allocated for client connections by setting the vbroker.se.iiop_ts.scm.iiop_ts.manager.connectionMax property.
Figure 12
The above figure shows the use of the thread-per-session policy. The Client application #1 establishes a connection with the object implementation. A separate connection exists between Client application #2 and the object implementation. When a request comes in to the object implementation from Client application #1, a worker thread handles the request. When a request from Client application #2 comes in, a different worker thread is assigned to handle this request.
Figure 13
In the above figure, a second request has come in to the object implementation from Client application #1. The same thread that handles request 1 will handle request 2. The thread blocks request 2 until it completes request 1. (With thread-per-session, requests from the same Client are not handled in parallel.) When request 1 has completed, the thread can handle request 2 from Client application #1. Multiple requests may come in from Client application #1. They are handled in the order that they come in; no additional threads are assigned to Client application #1.
Connection management
Overall, VisiBroker's connection management minimizes the number of client connections to the server. In other words there is only one connection per server process which is shared. All requests from a single client application are multiplexed over the same connection, even if they originate from different threads. Additionally, released client connections are recycled for subsequent reconnects to the same server, eliminating the need for clients to incur the overhead of new connections to the server.
In the following scenario, a client application is bound to two objects in the server process. Each bind() shares a common connection to the server process, even though the bind() is for a different object in the server process.
Figure 14
The following figure shows the connections for a client using multiple threads that has several threads bound to an object on the server.
Figure 15
As the above figure shows, all invocations from all threads are serviced by the same connection. For that scenario, the most efficient multi threading model to use is the thread pool model. If the thread-per-session model is used in this scenario, only one thread on the server will be allocated to service all requests from all threads in the client application, which could easily result in poor performance.
The maximum number of connections to a server, or from a client, can be configured. Inactive connections will be recycled when the maximum is reached, ensuring resource conservation.
PeerConnectionCurrent Interface
On the server side, a client's host and the port details are obtainable by the use of a PeerConnectionCurrent interface. The PeerConnectionCurrent interface is defined as follows:
public interface PeerConnectionCurrent {
public abstract java.lang.String getPeerHost();
public abstract int getPeerPort();
};
The reference to PeerConnectionCurrent interface is obtained by a call to org.omg.CORBA.ORB.resolve_initial_references("PeerConnectionCurrent"). If the client and server are colocated, a call to getPeerHost will return the localhost address and getPeerPort will return "0" (zero).
The host address is returned as a dotted IP address string. The precondition for the use of PeerConnectionCurrent is that it can only be used from inside a request's invocation context. Outside the invocation context, a call to getPeerHost and getPeerPort raises a BAD_INV_ORDER exception.
PeerConnectionCurrent can be called from inside in the following ways:
1
2
All ServerRequestInterceptor intercept points except for postinvoke_postmarshal
However, if the ServantLocator is being used, then the PeerConnectionCurrent cannot be called from within the ServerRequestInterceptor preinvoke or the ServerRequestInterceptor receive_request_service_contexts and the ServantLocator preinvoke methods. Otherwise, this will result with a BAD_INV_ORDER exception.
The following code illustrates the use of the PeerConnectionCurrent:
import com.inprise.vbroker.orb.PeerConnectionCurrent;
public class SomeServantImpl extends SomeServantPOA {
public int method(String name) {
.....
// assuming "orb" is already initialized
try {
PeerConnectionCurrent conninfo=
(PeerConnectionCurrent) orb.resolve_initial_references("PeerConnectionCurrent");
System.out.println("Client's host="+conninfo.getPeerHost());
System.out.println("Client's port="+conninfo.getPeerPort());
}
catch (Exception e) {
e.printStackTrace();
}
.....
}
}
ServerEngines
Thread and connection management on the server side is performed by ServerEngines, which can consist of one or more Server Connection Managers (SCMs). An SCM is a collection of properties of the manager, listener, and dispatcher.
Defining a ServerEngine consists of specifying a set of properties in a properties file. For example, if on UNIX the property file called myprops.properties is in home directory, the command line is
prompt> vbj -DORBpropStorage=~/myprops.properties myServer
ServerEngine properties
vbroker.se.<srvr_eng_name>.scms=<srvr_connection_mngr_name1>,<srvr_connection_mngr_name2>
The set of Server Connection Managers associated with a ServerEngine is defined by this property. The name specified in the above property as the <svr_eng_name> is the name of the ServerEngine. The SCMs listed here will be the list of SCMs for the associated server engine. SCMs cannot be shared between ServerEngines. However, ServerEngines can be shared by multiple POAs.
The other properties are
vbroker.se.<srvr_eng_name>.host
The host property is the IP address for the server engine to listen for messages.
vbroker.se.<srvr_eng_name>.proxyHost
The proxyHost property specifies the proxy IP address to send to the client in the case where the server does not want to publish its real hostname.
Setting dispatch policies and properties
Each POA in a multi-threaded object server can choose between two dispatch models: thread-per-session or thread pool. You choose a dispatch policy by setting the dispatcher.type property of the ServerEngine.
vbroker.se.<srvr_eng_name>.scm.<srvr_connection_mngr_name>.dispatcher.type=
ThreadPool
vbroker.se.<srvr_eng_name>.scm.<srvr_connection_mngr_name>.dispatcher.type=
ThreadSession
For more information about these properties see , “Using POAs” and the VisiBroker Programmer's Reference.
Thread pool dispatch policy
ThreadPool (thread pooling) is the default dispatch policy when you create a POA without specifying the ServerEnginePolicy.
For ThreadPool, you can set the following properties:
This property sets a TPool server engine's maximum number of worker threads in the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property
vbroker.se.default.dispatcher.tp.threadMax=32
or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=32
sets the initial maximum worker thread limitation to 32 for the default TPool server engine. The default value of this property is unlimited (0). If there are no threads available in the pool and the maximum number of threads have already been created, the request is blocked until a thread currently in use has been released back into the pool.
This property sets a TPool server engine's minimum number of worker threads in the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property
vbroker.se.default.dispatcher.tp.threadMin=8
or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMin=8
sets the initial worker thread minimum number to 8 for the default TPool server engine. The default value of this property is 0 (no worker threads).
This property sets a TPool server engine's idle thread check interval. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property
vbroker.se.default.dispatcher.tp.threadMaxIdle=120
or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=120
sets the initial idle worker thread check interval to 120 seconds for the default TPool server engine. The default value of this property is 300 seconds. With this setting, the server engine will check the idle state of each worker thread every 120 seconds. If a worker thread has been idle across two consecutive checks, it will be recycled (terminated) at the second check. Therefore, the actual idle thread garbage collection time is between 120 to 240 seconds under the above setting, instead of exactly 120 seconds.
The ThreadPool dispatcher allows a “cooling time” to be set. A thread is said to be “hot” when the GIOP connection being served is potentially readable, either upon creation of the connection or upon the arrival of a request. After the cooling time (in seconds), the thread can be returned to the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the startup property
vbroker.se.default.dispatcher.tp.coolingTime=6
or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=6
sets the initial cooling time to 6 seconds for the default engine (or the IIOP TPool server engine).
This property is applicable to VisiBroker for Java under certain conditions. See “High scalability configuration for VisiBroker for Java (using Java NIO)” for details. The default value of this property in VisiBroker for Java is 0 (zero), which implies that a GIOP connection being serviced ceases to be “hot” unless a new request is immediately available for servicing. It is important that the value of coolingTime is not altered unless tests have indicated that a non-default value is beneficial to the performance of the application.
Note
We recommend that you use the vbroker.se.default.xxx.tp.xxx property when vbroker.se.default is set to iiop_tp. When using with ThreadSession, we recommend that you use the vbroker.se.iiop_ts.scm.iiop_ts.xxx property.
Thread-per-session dispatch policy
When using the ThreadSession as the dispatcher type, you must set the se.default property to iiop_ts.
vbroker.se.default=iiop_ts
Note
In thread-per-session, there are no threadMin, threadMax, threadMaxIdle, and coolingTime dispatcher properties. Only the Connection and Manager properties are valid properties for ThreadSession.
Coding considerations
All code within a server that implements the VisiBroker ORB object must be thread-safe. You must take special care when accessing a system-wide resource within an object implementation. For example, many database access methods are not thread-safe. Before your object implementation attempts to access such a resource, it must first lock access to the resource using a synchronized block.
If serialized access to an object is required, you need to create the POA on which this object is activated with the SINGLE_THREAD_MODEL value for the ThreadPolicy.
Setting connection management properties
The following properties are used to configure connection management. Properties whose names start with vbroker.se are server-side properties. The client side properties have their names starting with vbroker.ce.
Note
The command line options for VisiBroker 3.x backward-compatibility are less obvious in terms of whether they are client-side or server-side. However, the connection and thread management options that start with the -ORB prefix set the client-side options whereas the options with the -OA prefix are used for the server-side options. There are no common properties which are used for both client-side and server-side thread and connection management.
 
The distinction between client and server vanishes if callback or bidirectional GIOP is used.
This property sets the maximum allowable client connections to a server engine. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property
-Dvbroker.se.default.socket.manager.connectionMax=128
or
-Dvbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMax
=128
sets the initial maximum connection limitation on this server engine to 128. The default value of this property is unlimited (0 [zero]). When the server engine reaches this limitation, before accepting a new client connection, the server engine needs to reuse an idle connection. This is called connection swapping. When a new connection arrives at the server, it will try to detach the oldest unused connection. If all the connections are busy, the new connection will be dropped. The client may retry again until some timeout expires.
This property sets the maximum length of time an idle connection will remain open on a server engine. The property can be set statically on server startup or dynamically reconfigured using property API. For instance, the start up property
-Dvbroker.se.default.socket.manager.connectionMaxIdle
=300
or
-Dvbroker.se.iiop_tp.scm.iiop_tp.manager.
connectionMaxIdle=300
sets the initial idle connection maximum lifetime to 300 seconds. The default value of this property is 0 (unlimited). When a client connection has been idle longer than this value, it becomes a candidate for garbage collection.
Specifies the maximum number of the total connections within a client. The default value of zero means that the client does not try to close any of the old active or cached connections. If a new client connection will result in exceeding the limit set by this property, VisiBroker for Java will try to release one of the cached connections. If there are no cached connections, it will try to close the oldest idle connection. If both of them fail, the CORBA::NO_RESOURCE exception will result.
An NIO connection may time out waiting for a still connected client to write further bytes because it has exceeded the value configured in the property vbroker.orb.serverRecvTimeout. You have the option whether or not to disconnect the client.
If vbroker.orb.serverRecvTimeoutDisconnects is set to true, the client's connection will be closed if it times out. This is the default if the property is not explicitly set and matches the behavior of the SSL NIO SCM.
If this property is set to false, the connection server thread will unblock after the timeout, allowing other waiting server threads to be served from the selector pool. After some time it may then attempt to read the remainder of the message from the client's connection again, when again it may hold a selector for up to serverRecvTimeout during the attempt.
Valid values for applicable properties
The following properties have a fixed set or range of valid values:
Currently, Pool is the only supported type.
In the following properties, xxx is the server engine name and yyy is the server connection manager name:
Socket_nio is the only other permissable value for this property.
You can also use the value SSL for security.
The other possible values are ThreadSession and MainThread.
The default value is 0 (zero) , and the maximum value is 10, so a value greater than 10 will be clamped to 10. In VisiBroker for Java, this property is applicable only if the Server Connection Manager has a manager type of Socket_nio. See “High scalability configuration for VisiBroker for Java (using Java NIO)” for details.
Effects of property changes
The effect of a change in a property value depends on the actions associated with the properties. Most of the actions are directly or indirectly related to the utilization of system resources. The availability and restrictions of the system resources to the CORBA application vary depending on the system and the nature of the application.
For instance, decreasing the garbage collector timer may increase the system activities, as the garbage collector will run more frequently. On the other hand, increasing its value means the idle threads will remain in system unclaimed for longer periods of time.
Dynamically alterable properties
The following properties can be changed dynamically and the effect will be immediate unless stated otherwise:
vbroker.ce.iiop.ccm.connectionMax=0
vbroker.ce.iiop.ccm.connectionMaxIdle=360
vbroker.ce.iiop.connection.tcpNoDelay=false
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMax=0
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=0
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMin=0
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=100
The new dispatcher threadMax properties will be reflected after the next garbage collector run.
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=300
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=3
Determining whether property value changes take effect
For this purpose, the server manager needs to be enabled, using the property vbroker.orb.enableServerManager=true, and the properties can be obtained through the server manager query either through the Console or through a command-line utility.
Impact of changing property values
It is very difficult to determine the impact of changing the value of a property to something other than the default. For thread and connection limits, the available system resources vary depending on the machine configuration and the number of other processes running. The setting of properties allows performance tuning for a given system.
High scalability configuration for VisiBroker for Java (using Java NIO)
The Java NIO package, available in J2SE 1.4, allows servers to handle multiple connections efficiently, without having to dedicate a thread per connection. This allows servers to service a large number of client connections with fewer threads, translating to higher scalability. VisiBroker for Java servers can be configured to harness Java NIO technology. Servers using the ThreadPool policy can use Java NIO by setting the manager type to Socket_nio instead of Socket. For example,
vbroker.se.iiop_tp.scm.iiop_tp.manager.type=Socket_nio
This feature should be used in combination with the threadMax property, which is used to limit the number of threads in the thread pool that are available for dispatching requests (i.e., processing invocations). When the manager type is Socket_nio, the number of threads in the thread pool will not increase (beyond the number specified as threadMax) proportionate to the number of connections being serviced. This is possible because here the necessity to have a thread per connection does not exist.
Please note that the thread per connection model (which is the default for the VisiBroker for Java thread pool) is expected to outperform the NIO based model for servers where the number of connections is relatively small (i.e., not of the order of hundreds of connections). It is advisable to run tests to decide on the appropriate model given the typical load conditions for an application.
Servers using J2SE 1.4 or above will be able to use this feature. Currently, clients based on VisiBroker for Java do not benefit from the ORB's usage of Java NIO.
The coolingTime property is effective in VisiBroker for Java when NIO based dispatch is enabled. See “Thread pool dispatch policy” for details.
Note that if the property vbroker.se.iiop_tp.scm.iiop_tp.listener.useSelectorPool is set to true, then each IIOP NIO Listener in the server will be constrained to consume a maximum of vbroker.se.iiop_tp.scm.iiop_tp.listener.selectorMax instances of java.nio.channels.Selector at any given time. If that property is not set, or is set to false, the number of Selectors in use will continue to be unbounded and to be some function of the number of concurrent connections for non-SSL NIO connections.
Garbage collection
The VisiBroker for Java ORB performs automatic garbage collection of various resources other than the memory. The garbage collection of the memory is performed by the Java Virtual machine. Various properties are provided to control the garbage collection period. In addition, resources like threads and connections define timeout properties that control the collection of these resources.
How ORB garbage collection works
The ORB garbage collector thread is a normal priority thread. After the expiration of timeout period (specified by the property vbroker.orb.gcTimeout), it wakes up and collects all the resources that are idle and no longer in use. Classes interested in getting collected register themselves with the garbage collector. Such classes are called collectables. Prominent examples of collectables are threads and connections. Other examples include timeout on various caches like GateKeeper's cache, for example. Most of the collectables null out or properly release the resources (such as closing the connection or terminating a thread's run method) held by them when they are collected. These resources are later reclaimed by the Java garbage collector.
Note
The ORB garbage collector is an internal service and is not exposed to the user.
Properties related to ORB garbage collection
The main property that controls the garbage collection period is vbroker.orb.gcTimeout. The timeout value is in seconds and the default value is 30 seconds.
Threads and connections define properties for idle timeout. For example, the thread pool dispatcher defines the following property:
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle
The value is in seconds and default value is 300 seconds after which the thread is removed from the thread pool. Similarly, the default Server Connection Manager (iiop_tp) defines the following idle timeout property for connections.
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle
The value is in seconds and default value is 0(zero) which means a connection never gets closed no matter how long it remains idle. However, if the connection gets dropped, the ORB removes all the references to it and its resources are later collected by Java garbage collector. The ORB garbage collector will only collect connections whose connectionMaxIdle property is set to a non-zero value.
The various timeout properties and the vbroker.orb.gcTimeout property have a subtle relationship. For example, suppose following properties are specified:
vbroker.orb.gcTimeout=10
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=5
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=5
Here the garbage collection timeout period is set to 10 seconds whereas thread and connection timeouts are 5 seconds. The figure below illustrates how these properties interact. Here we have shown a thread, T1, and a connection, C1, that have gone idle and are then collected.
Note
Although the ORB garbage collector is shown here as running exactly after ten seconds, in practice this may not be true depending on when the JVM schedules the garbage collector (GC) thread.
Figure 16
Even though T1 and C1 are eligible for collection, they are collected only when the ORB garbage collector runs. Until then they remain in the timed-out state.