1 Introduction
2 Using Qpid Dispatch
3 Addressing
- 3.1 Routing patterns
- 3.2 Routing mechanisms
  - 3.2.1 Message routing
  - 3.2.2 Link routing
4 AMQP Mapping
5 The qdrouter management schema
6 Manual page qdrouterd.8
7 Manual page qdstat.8
8 Manual page qdmanage.8
9 Manual page qdrouterd.conf.5

1 Introduction

1.1 Overview

The Dispatch router is an AMQP message message router that provides advanced interconnect capabilities. It allows flexible routing of messages between any AMQP-enabled endpoints, whether they be clients, servers, brokers or any other entity that can send or receive standard AMQP messages.

A messaging client can make a single AMQP connection into a messaging bus built of Dispatch routers and, over that connection, exchange messages with one or more message brokers, and at the same time exchange messages directly with other endpoints without involving a broker at all.

The router is an intermediary for messages but it is not a broker. It does not take responsibility for messages. It will, however, propagate settlement and disposition across a network such that delivery guarantees are met. In other words: the router network will deliver the message, possibly via several intermediate routers, and it will route the acknowledgement of that message by the ultimate reciever back across the same path. This means that responsibility for the message is transfered from the original sender to the ultimate receiver as if they were directly connected. However this is done via a flexible network that allows highly configurable routing of the message transparent to both sender and receiver.

There are some patterns where this enables "brokerless messaging" approaches that are preferable to brokered approaches. In other cases a broker is essential (in particular where you need the separation of responsibility and/or the buffering provided by store-and-forward) but a dispatch network can still be useful to tie brokers and clients together into patterns that are difficult with a single broker.

For a "brokerless" example, consider the common brokered implementation of the request-response pattern, a client puts a request on a queue and then waits for a reply on another queue. In this case the broker can be a hindrance - the client may want to know immediatly if there is nobody to serve the request, but typically it can only wait for a timeout to discover this. With a dispatch network, the client can be informed immediately if its message cannot be delivered because nobody is listening. When the client receives acknowledgement of the request it knows not just that it is sitting on a queue, but that it has actually been received by the server.

For an exampe of using dispatch to enhance the use of brokers, consider using an array of brokers to implement a scalable distributed work queue. A dispatch network can make this appear as a single queue, with senders publishing to a single address and receivers subscribing to a single address. The dispatch network can distribute work to any broker in the array and collect work from any broker for any receiver. Brokers can be shut down or added without affecting clients. This elegantly solves the common difficult of "stuck messages" when implementing this pattern with brokers alone. If a receiver is connected to a broker that has no messages, but there are messages on another broker, you have to somehow transfer them or leave them "stuck". With a dispatch network, all the receivers are connected to all the brokers. If there is a message anywhere it can be delivered to any receiver.

The router is meant to be deployed in topologies of multiple routers, preferably with redundant paths. It uses link-state routing protocols and algorithms (similar to OSPF or IS-IS from the networking world) to calculate the best path from every point to every other point and to recover quickly from failures. It does not need to use clustering for high availability; rather, it relies on redundant paths to provide continued connectivity in the face of system or network failure. Because it never takes responsibility for messages it is effectively stateless, messages not delivered to their final destination will not be acknowledged to the sender and therefore the sender can re-send such messages if it is disconnected from the network.

1.2 Benefits

Simplifies connectivity
An endpoint can do all of its messaging through a single transport connection
Avoid opening holes in firewalls for incoming connections
Simplifies reliability
Reliability and availability are provided using redundant topology, not server clustering
Reliable end-to-end messaging without persistent stores
Use a message broker only when you need store-and-forward semantics

1.3 Features

Supports arbitrary topology - no restrictions on redundancy
Automatic route computation - adjusts quickly to changes in topology
Cost-based route computation
Rich addressing semantics
Security

2 Using Qpid Dispatch

2.1 Configuration

The default configuration file is installed in install-prefix/etc/qpid/qdrouterd.conf. This configuration file will cause the router to run in standalone mode, listening on the standard AMQP port (5672). Dispatch Router looks for the configuration file in the installed location by default. If you wish to use a different path, the "-c" command line option will instruct Dispatch Router as to which configuration to load.

To run the router, invoke the executable: o $ qdrouterd [-c my-config-file]

For more details of the configuration file see the qdrouterd.conf(5) man page.

2.2 Client Compatibility

Dispatch Router should, in theory, work with any client that is compatible with AMQP 1.0. The following clients have been tested:

Client	Notes
qpid::messaging	The Qpid messaging clients work with Dispatch Router as long as they are configured to use the 1.0 version of the protocol. To enable AMQP 1.0 in the C++ client, use the {protocol:amqp1.0} connection option.
Proton Messenger	Messenger works with Dispatch Router.

2.3 Tools

2.3.1 qdstat

qdstat is a command line tool that lets you view the status of a Dispatch Router. The following options are useful for seeing that the router is doing:

Option	Description
-l	Print a list of AMQP links attached to the router. Links are unidirectional. Outgoing links are usually associated with a subscription address. The tool distinguishes between endpoint links and router links. Endpoint links are attached to clients using the router. Router links are attached to other routers in a network of routers.
-a	Print a list of addresses known to the router.
-n	Print a list of known routers in the network.
-c	Print a list of connections to the router.

For complete details see the qdstat(8) man page and the output of qdstat --help.

2.3.2 qdmanage

qdmanage is a general-purpose AMQP management client that allows you to not only view but modify the configuration of a running dispatch router.

For example you can query all the connection entities in the route r $ qdrouterd query --type connection

To enable logging debug and higher level messages by default: $ qdrouter update log/DEFAULT enable=debug+

In fact, everything that can be configured in the configuration file can also be created in a running router via management. For example to create a new listener in a running router: $ qdrouter create type=listener port=5555

Now you can connect to port 5555, for exampple $ qdrouterd query -b localhost:5555 --type listener

For complete details see the qdmanage(8) man page and the output of qdmanage --help. Also for details of what can be configured see the qdrouterd.conf(5) man page.

2.4 Features and Examples

2.4.1 Standalone and Interior Modes

The router can operate stand-alone or as a node in a network of routers. The mode is configured in the router section of the configuration file. In stand-alone mode, the router does not attempt to collaborate with any other routers and only routes messages among directly connected endpoints.

If your router is running in stand-alone mode, qdstat -a will look like the following:

$ qdstat -a
Router Addresses
  class  address      in-proc  local  remote  in  out  thru  to-proc  from-proc
  ===============================================================================
  local  $management  Y        0      0       1   0    0     1        0
  local  temp.AY81ga           1      0       0   0    0     0        0

Note that there are two known addresses. $management is the address of the router's embedded management agent. temp.AY81ga is the temporary reply-to address of the qdstat client making requests to the agent.

If you change the mode to interior and restart the processs, the same command will yield two additional addresses which are used for inter-router communication:

$ qdstat -a
Router Addresses
  class  address      in-proc  local  remote  in  out  thru  to-proc  from-proc
  ===============================================================================
  local  $management  Y        0      0       1   0    0     1        0
  local  qdhello      Y        0      0       0   0    0     0        3
  local  qdrouter     Y        0      0       0   0    0     0        1
  local  temp.khOpGb           1      0       0   0    0     0        0

2.4.2 Mobile Subscribers

The term "mobile subscriber" simply refers to the fact that a client may connect to the router and subscribe to an address to receive messages sent to that address. No matter where in the network the subscriber attaches, the messages will be routed to the appropriate destination.

To illustrate a subscription on a stand-alone router, you can use the examples that are provided with Qpid Proton. Using the recv.py example receiver:

$ recv.py amqp://0.0.0.0/my-address

This command creates a receiving link subscribed to the specified address. To verify the subscription:

$ qdstat -a
Router Addresses
  class   address      in-proc  local  remote  in  out  thru  to-proc  from-proc
  ================================================================================
  local   $management  Y        0      0       1   0    0     1        0
  mobile  my-address            1      0       0   0    0     0        0
  local   temp.fDt8_a           1      0       0   0    0     0        0

You can then, in a separate command window, run a sender to produce messages to that address:

$ send.py -a amqp://0.0.0.0/my-address

2.4.3 Dynamic Reply-To

Dynamic reply-to can be used to obtain a reply-to address that routes back to a client's receiving link regardless of how many hops it has to take to get there. To illustrate this feature, see below a simple program (written in C++ against the qpid::messaging API) that queries the management agent of the attached router for a list of other known routers' management addresses.

#include <qpid/messaging/Address.h>
#include <qpid/messaging/Connection.h>
#include <qpid/messaging/Message.h>
#include <qpid/messaging/Receiver.h>
#include <qpid/messaging/Sender.h>
#include <qpid/messaging/Session.h>

using namespace qpid::messaging;
using namespace qpid::types;

using std::stringstream;
using std::string;

int main() {
    const char* url = "amqp:tcp:127.0.0.1:5672";
    std::string connectionOptions = "{protocol:amqp1.0}";

    Connection connection(url, connectionOptions);
    connection.open();
    Session session = connection.createSession();
    Sender sender = session.createSender("mgmt");

    // create reply receiver and get the reply-to address
    Receiver receiver = session.createReceiver("#");
    Address responseAddress = receiver.getAddress();

    Message request;
    request.setReplyTo(responseAddress);
    request.setProperty("x-amqp-to", "amqp:/_local/$management");
    request.setProperty("operation", "DISCOVER-MGMT-NODES");
    request.setProperty("type", "org.amqp.management");
    request.setProperty("name, "self");

    sender.send(request);
    Message response = receiver.fetch();
    Variant content(response.getContentObject());
    std::cout << "Response: " << content << std::endl << std::endl;

    connection.close();
}

The equivalent program written in Python against the Proton Messenger API:

from proton import Messenger, Message

def main():
    host = "0.0.0.0:5672"

    messenger = Messenger()
    messenger.start()
    messenger.route("amqp:/*", "amqp://%s/$1" % host)
    reply_subscription = messenger.subscribe("amqp:/#")
    reply_address = reply_subscription.address

    request  = Message()
    response = Message()

    request.address = "amqp:/_local/$management"
    request.reply_to = reply_address
    request.properties = {u'operation' : u'DISCOVER-MGMT-NODES',
                          u'type'      : u'org.amqp.management',
                          u'name'      : u'self'}

    messenger.put(request)
    messenger.send()
    messenger.recv()
    messenger.get(response)

    print "Response: %r" % response.body

    messenger.stop()

main()

2.5 Known Issues and Limitations

This is an early test release. It is expected that users will find bugs and other various instabilities. The main goal of this release is to prove that the process can be run and that users can demonstrate basic functionality as described in this document. Nevertheless, the following are known issues with the 0.1 release:

Subscriber addresses are not always cleaned up after a consumer disconnects. See https://issues.apache.org/jira/browse/QPID-4964.
Dispatch Router does not currently use the target address of a client's sender link to route messages. It only looks at the "to" field in the message's headers. See https://issues.apache.org/jira/browse/QPID-5175.
All subscription sources are treated as multicast addresses. There is currently no facility for provisioning different types of addresses. Multicast means that if there are multiple subscribers to the same address, they will all receive a copy of each message sent to that address.
SSL connectors and listeners are supported but very lightly (and not recently) tested.
SASL authentication is not currently integrated into any authentication framework. Use ANONYMOUS for testing.

3 Addressing

AMQP addresses are used to control the flow of messages across a network of routers. Addresses are used in a number of different places in the AMQP 1.0 protocol. They can be used in a specific message in the to and reply-to fields of a message's properties. They are also used during the creation of links in the address field of a source or a target.

Addresses designate various kinds of entities in a messaging network:

Endpoint processes that consume data or offer a service
Topics that match multiple consumers to multiple producers
Entities within a messaging broker:
Queues
Durable Topics
Exchanges

The syntax of an AMQP address is opaque as far as the router network is concerned. A syntactical structure may be used by the administrator that creates addresses, but the router treats them as opaque strings. Routers consider addresses to be mobile such that any address may be directly connected to any router in a network and may move around the topology. In cases where messages are broadcast to or balanced across multiple consumers, an address may be connected to multiple routers in the network.

Addresses have semantics associated with them. When an address is created in the network, it is assigned a set of semantics (and access rules) during a process called provisioning. The semantics of an address control how routers behave when they see the address being used.

Address semantics include the following considerations:

Routing pattern - direct, multicast, balanced
Routing mechanism - message routed, link routed
Undeliverable action - drop, hold and retry, redirect
Reliability - N destinations, etc.

3.1 Routing patterns

Routing patterns constrain the paths that a message can take across a network.

Pattern	Description
Direct	Direct routing allows for only one consumer to use an address at a time. Messages (or links) follow the lowest cost path across the network from the sender to the one receiver.
Multicast	Multicast routing allows multiple consumers to use the same address at the same time. Messages are routed such that each consumer receives a copy of the message.
Balanced	Balanced routing also allows multiple consumers to use the same address. In this case, messages (or links) are routed to exactly one of the consumers, and the network attempts to balance the traffic load across the set of consumers using the same address.

3.2 Routing mechanisms

The fact that addresses can be used in different ways suggests that message routing can be accomplished in different ways. Before going into the specifics of the different routing mechanisms, it would be good to first define what is meant by the term routing:

In a network built of multiple routers connected by connections (i.e., nodes and edges in a graph), routing determines which connection to use to send a message directly to its destination or one step closer to its destination.

Each router serves as the terminus of a collection of incoming and outgoing links. Some of the links are designated for message routing, and others are designated for link routing. In both cases, the links either connect directly to endpoints that produce and consume messages, or they connect to other routers in the network along previously established connections.

3.2.1 Message routing

Message routing occurs upon delivery of a message and is done based on the address in the message's to field.

When a delivery arrives on an incoming message-routing link, the router extracts the address from the delivered message's to field and looks the address up in its routing table. The lookup results in zero or more outgoing links onto which the message shall be resent.

Delivery	Handling
pre-settled	If the arriving delivery is pre-settled (i.e., fire and forget), the incoming delivery shall be settled by the router, and the outgoing deliveries shall also be pre-settled. In other words, the pre-settled nature of the message delivery is propagated across the network to the message's destination.
unsettled	Unsettled delivery is also propagated across the network. Because unsettled delivery records cannot be discarded, the router tracks the incoming deliveries and keeps the association of the incoming deliveries to the resulting outgoing deliveries. This kept association allows the router to continue to propagate changes in delivery state (settlement and disposition) back and forth along the path which the message traveled.

pre-settled

If the arriving delivery is pre-settled (i.e., fire and forget), the incoming delivery shall be settled by the router, and the outgoing deliveries shall also be pre-settled. In other words, the pre-settled nature of the message delivery is propagated across the network to the message's destination.

unsettled

Unsettled delivery is also propagated across the network. Because unsettled delivery records cannot be discarded, the router tracks the incoming deliveries and keeps the association of the incoming deliveries to the resulting outgoing deliveries. This kept association allows the router to continue to propagate changes in delivery state (settlement and disposition) back and forth along the path which the message traveled.

3.2.2 Link routing

Link routing occurs when a new link is attached to the router across one of its AMQP connections. It is done based on the target.address field of an inbound link and the source.address field of an outbound link.

Link routing uses the same routing table that message routing uses. The difference is that the routing occurs during the link-attach operation, and link attaches are propagated along the appropriate path to the destination. What results is a chain of links, connected end-to-end, from source to destination. It is similar to a virtual circuit in a telecom system.

Each router in the chain holds pairs of link termini that are tied together. The router then simply exchanges all deliveries, delivery state changes, and link state changes between the two termini.

The endpoints that use the link chain do not see any difference in behavior between a link chain and a single point-to-point link. All of the features available in the link protocol (flow control, transactional delivery, etc.) are available over a routed link-chain.

4 AMQP Mapping

Dispatch Router is an AMQP router and as such, it provides extensions, codepoints, and semantics for routing over AMQP. This page documents the details of Dispatch Router's use of AMQP.

4.1 Message Annotations

The following Message Annotation fields are defined by Dispatch Router:

Field	Type	Description
x-opt-qd.ingress	string	The identity of the ingress router for a message-routed message. The ingress router is the first router encountered by a transiting message. The router will, if this field is present, leave it unaltered. If the field is not present, the router shall insert the field with its own identity.
x-opt-qd.trace	list of string	The list of routers through which this message-routed message has transited. If this field is not present, the router shall do nothing. If the field is present, the router shall append its own identity to the end of the list.
x-opt-qd.to	string	To-Override for message-routed messages. If this field is present, the address in this field shall be used for routing in lieu of the to field in the message properties. A router may append, remove, or modify this annotation field depending on the policy in place for routing the message.
x-opt-qd.class	string	Message class. This is used to allow the router to provide separate paths for different classes of traffic.

4.2 Source/Target Capabilities

The following Capability values are used in Sources and Targets.

Capability	Description
qd.router	This capability is added to sources and targets that are used for inter-router message exchange.

4.3 Addresses and Address Formats

The following AMQP addresses and address patterns are used within Dispatch Router.

4.3.1 Address Patterns

Pattern	Description
`_local/<addr>`	An address that references a locally attached endpoint. Messages using this address pattern shall not be routed over more than one link.
`_topo/<area>/<router>/<addr>`	An address that references an endpoint attached to a specific router node in the network topology. Messages with addresses that follow this pattern shall be routed along the shortest path to the specified router. Note that addresses of this form are a-priori routable in that the address itself contains enough information to route the message to its destination.
`<addr>`	A mobile address. An address of this format represents an endpoint or a set of distinct endpoints that are attached to the network in arbitrary locations. It is the responsibility of the router network to determine which router nodes are valid destinations for mobile addresses.

4.3.2 Supported Addresses

Address	Description
`_local/$management`	The management agent on the attached router/container. This address would be used by an endpoint that is a management client/console/tool wishing to access management data from the attached container.
`_topo/0/Router.E/agent`	The management agent at Router.E in area 0. This address would be used by a management client wishing to access management data from a specific container that is reachable within the network.
`_local/qdhello`	The router entity in each of the connected routers. This address is used to communicate with neighbor routers and is exclusively for the HELLO discovery protocol.
`_local/qdrouter`	The router entity in each of the connected routers. This address is used by a router to communicate with other routers in the network.
`_topo/0/Router.E/qdxrouter`	The router entity at the specifically indicated router. This address form is used by a router to communicate with a specific router that may or may not be a neighbor.

4.4 Implementation of the AMQP Management Specification

Qpid Dispatch is manageable remotely via AMQP. It is compliant with the emerging AMQP Management specification (draft 9).

Differences from the specification:

The "name" attribute is not required when an entity is created. If not supplied it will be set to the same value as the system-generated "identity" attribute. Otherwise it is treated as per the standard.
The REGISTER operation is not implementd. The router has its own mechansm to discover peers that does not require this operation.
The DEREGISTER operation is not implementd. The router has its own mechansm to discover peers that does not require this operation.

5 The qdrouter management schema

The schema qdrouterd.json is a JSON format file that defines annotations and entity types of the Qpid Dispatch Router management model. The model is based on the AMQP management specification.

The schema is a JSON map with the following keys:

"description": documentation string for the schema
"prefix": Prefix prepended to schema names when they are exposed to AMQP management clients.
"annotations": map of annotation names to definitions (see below)
"entityTypes": map of entity type names to definitions (see below)

Annotation and entity type definition maps have the following keys:

"description": documentation string.
"operations": list of allowed operation names.
"attributes": map of attribute names to attribute definitions (see below)

Entity type definitions also have these fields:

"extends": Name of base type. The new type includes operations and attributes from the base type.
"annotations": List of annotation names. The new type includes operations and attributes from all the annotations.

Attribute definition maps have the following fields:

"type": one of the following:
"String": a unicode string value.
"Integer": an integer value.
"Boolean": a true/false value.
"default": a default can be a literal value or a reference to another attribute in the form $attributeName.

There is the following hierarchy among entity types:

entity: The base of all entity types. - configurationEntity: base for all types that hold configuration information.

Configuration information is supplied in advance and expresse intent. For example "I want the router to listen on port N". All the entities that can be used in the configuration file extend configurationEntity.

operationalEntity: base for all types that hold operational information.

Operational information reflects the actual current state of the router. For example, "how many addresses are presently active?" All the entities queried by the qdstat tool extend operationalEntity.

The two types are often related. For example listener and connector extend configurationEntity, they express the intent to make or receive connections. connection extends operationalEntity, it holds information the actual connection status.

The rest of this section provides the schema documentation in readable format.

5.1 Annotations

5.1.1 `addrPort`

Attributes for internet address and port.

Used by listener, connector.

addr (String, default=0.0.0.0): Host address: ipv4 or ipv6 literal or a host name.
port (String, default=amqp): Port number or symbolic service name.

5.1.2 `saslMechanisms`

Attribute for a list of SASL mechanisms.

Used by listener, connector.

saslMechanisms (String, required): Comma separated list of accepted SASL authentication mechanisms.

5.1.3 `connectionRole`

Attribute for the role of a connection.

Used by listener, connector.

role (One of [normal, inter-router, on-demand], default=normal): The role of an established connection. In the normal role, the connection is assumed to be used for AMQP clients that are doing normal message delivery over the connection. In the inter-router role, the connection is assumed to be to another router in the network. Inter-router discovery and routing protocols can only be used over interRouter connections.

5.1.4 `sslProfile`

Attributes for setting TLS/SSL configuration for connections.

Used by listener, connector.

certDb (String): The path to the database that contains the public certificates of trusted certificate authorities (CAs).
certFile (String): The path to the file containing the PEM-formatted public certificate to be used on the local end of any connections using this profile.
keyFile (String): The path to the file containing the PEM-formatted private key for the above certificate.
passwordFile (String): If the above private key is password protected, this is the path to a file containing the password that unlocks the certificate key.
password (String): An alternative to storing the password in a file referenced by passwordFile is to supply the password right here in the configuration file. This option can be used by supplying the password in the 'password' option. Don't use both password and passwordFile in the same profile.

5.2 Base Entity Type

5.2.1 `entity`

Base entity type for all entities.

name (String, unique): Unique name, can be changed.
identity (String, unique): Unique identity, will not change.
type (String, required): Management entity type.

5.3 Configuration Entities

5.3.1 `configurationEntity`

Base type for entities containing configuration information.

Extends: entity (name, identity, type).