I have read the following white papers on how to implement ordered delivery
and some of the challenges around it. I am currently tasked to solve the
ordered delivery problem and I need some advice/recommendations on my
approach.
Sequential FIFO Message Processing and Delivery in BizTalk Server 2004 by
Arno Harteveld & Erik Leaseburg
http://www.microsoft.com/downloads/details.aspx?FamilyID=F4FF7AFC-81A...
Implementing FIFO processing with BizTalk 2006 by Lea Monson
http://biztalkhotrod.com/Do***ents/Hotrod1Q32007%20Final.pdf
-----------
Scenario:
-----------
In my scenario, BizTalk will be an ESB that will route data from one
system
to multiple other systems.
System A: Maintains and owns service tickets
System B: Also maintains service tickets but keeps referential copies
System C: Also maintains service tickets but keeps referential copies
System D: Also maintains service tickets but keeps referential copies
1) A new service ticket is created/updated/closed in System A
2) System A sends this create/update/close ticket to BizTalk in System A's
proprietary format via a SOAP Receive ****t (WS)
3) BizTalk transforms the message to a canonical message format
4) BizTalk evaluates routing rules and determines that systems B,C,D need
to
receive copies of the new/udpated service ticket
4) BizTalk sends copies of the message to systems B,C,D in their
respective
proprietary format (basically a outbound transform is performed)
5) Systems B,C,D each return Acks or Nacks indicating a successful or
failed
operation
---------------
Problem (s):
---------------
System A is acting in a near real-time fa****on and sending
create/update/close tickets as they happen. Create/Update/Close are
represented with the same schema but they have a operation field that
indicates if it is a create/update/close.
1) What if an update is received before a create?
2) What if 2 updates come out of sequence?
3) What if a close is received before a create?
-----------------------------------------------------
MSMQ Ordered Delivery Solution (Approach 1)
-----------------------------------------------------
System A will not send messages via SOAP-HTTP. Instead we will require
System A to send in messages into a MSMQ queue. Messages in the queue
will
be placed in the correct order by System A. A queue with messages might
look
like the following:
Operation/Service Ticket Number
-----------------------------------------
END| X/2, X/1, U/1, U/2, U/1, C/2, C/1 | BEG
-----------------------------------------
C - Create
U - Update
X - Close
Ordered delivery will be turned on in the MSMQ receive location. A
sequential convoy orchestration will be designed with a logical receive
****t
that has ordered delivery set to true. The initial and following
correlations will be based on Receive ****t and Service Ticket Number.
This
will activate a single orchestration for the lifetime of a service ticket
number. In the above example, we will have two orch instances. First
will
process all creates and updates for service ticket #1 and the second will
process creates and updates for service ticket #2. Messages will be sent
in
a loop. Basically, this is the FIFO orchestration from the referenced
BTS04
white paper.
Business is also ok with intermediate updates that fail. The
orchestration
can continue processing the next update and log the update that failed.
Create must complete before processing subsequent updates.
------------------------------------------
Problems with Solution (Approach 1)
------------------------------------------
The nature of the business dictates that there is no stop condition for
receiving further updates. A create serivce ticket can come today, a few
updates can come tomorrow, and the close may not come for 2 months! The
business is also 24/7 so we can't say that Orch can be shutdown at
midnight
or after 12 hours. Based on the expected load, if I use the time-out
approach in the orch, I am already expecting to end up with zombie
messages.
Is there any way to solve this problem when there is no way to get a
proactive control message from System A that it is done sending?
Can I subscribe to the zombie messages and reprocess them afterwards?
Instead of a singleton orch, can an orch be designed to process each
message
and after receiving the ack, spawn another orch to process the next
message?
Essentially, a new orch instance is spawned after processing the previous.
-----------------------------------------------------
Re-Sequencer Pattern Solution (Approach 2)
-----------------------------------------------------
System A will send messages via SOAP-HTTP or any other adapter. Each
message must contain a sequence number (1,2,3,etc) and also a flag
indicating
LastMessage (true/false).
In BizTalk, I will implement the re-sequencer pattern and re-sequence the
messages based on the sequence number. Correlation set will be setup
based
on service ticket number. This will result in one singleton re-sequencer
per
service ticket number.
-----------------------------------------
Problems with Solution (Approach 2)
------------------------------------------
The nature of the business dictates that there is no stop condition for
receiving further updates. There is no way for System A to send BizTalk a
message which has LastMessage=true.
I AM LOOKING FOR SUGGESTIONS AND IDEAS ON HOW SOME OF THESE PROBLEMS CAN
BE
ADDRESSED WITH MY CURRENT SOLUTIONS ABOVE. IF YOU HAVE BETTER IDEAS ON
HOW
TO DEAL WITH THIS PROBLEM, I AM OPEN TO THOSE AS WELL. ANY HELP WILL BE
GREATLY APPRECIATED.
|
