I am having a problem with the Flat File Assembler which seems to be
incorrectly adding escape characters to the flat file do***ents it
creates.
It seems that where a string exists in a field that contains the same
characters as any of the Record Tag identifiers used in the do***ent then
that field value will have the escape character added in the created flat
file output.
To test my theory I have created a flat file schema for the do***ent type
that I want to be able to parse and produce. I then created both a receive
pipeline and a send pipeline that use the schema as the do***ent schema in
a
Flat File Disassembler and Flat File Assembler in the created receive and
send pipelines.
Everything worked as expected until I tried to validate a do***ent that
had
the + delimiter within one of the string fields. For example the + used in
the string MERANTI HNG?+STRIKER S/SET as shown below.
LIN+1363+3+9314916206526:EN'
IMD+F++:::MERANTI HNG?+STRIKER S/SET:2100x98x19'
PRI+AAA:19.93000:CA:INV:1:EA'
To overcome this problem I set the Escape character on the parent record
to
? and was then able to successfully validate the do***ent.
However when I then received that same test do***ent through the receive
pipeline that I created and then sent the xml that was produced through
the
send pipeline I created, I expected that the flat file that would be
produced
would be the same as the original input test file. However this was not
the
case. The differences were all caused by the escape character ? being
added
in places that didn’t really make any sense.
For example in the original input file the following record existed.
LIN+1055+3+9314916035683:EN'
IMD+F++:::GLZ LATTICE SPRING BARWON:0x0x0'
PRI+AAA:130.50000:CA:INV:1:EA'
This record was produced as the following the output file.
LIN+1055+3+9314916035683:EN'
IMD+F++:::GLZ LATTICE S?P?R?ING BARWON:0x0x0'
PRI+AAA:130.50000:CA:INV:1:EA'
And the following record in the original input file
LIN+640+3+9314916075993:EN'
IMD+F++:::REFLECTIONS DOOR ARCH TOP:2040x820x35'
PRI+AAA:196.04000:CA:INV:1:EA'
Was produced as follows in the output file
LIN+640+3+9314916075993:EN'
IMD+F++:::REFLECTIONS DOOR ARCH TO?P:2040x820x35'
PRI+AAA:196.04000:CA:INV:1:EA'
I was expecting that only the delimiter characters (‘ + and : ) would be
escaped but as you can see this wasn’t the case.
After further investigation I found that it was the characters that I had
used in the Tag Identifiers that were being escaped in addition to the
delimiters. To prove this I changed the schema and test input do***ent and
created new pipelines so that instead of using the Tag identifiers LIN,
IMD
and PRI I used ZZX, ZZY, and ZZZ respectively.
After completing the same exercise of sending the modified test do***ent
through the receive and send pipelines I found that the result was as I
expected with one exception. Where a single quote existed in a string
field
and it had been escaped in the input do***ent I expected that it would
also
be escaped in the output do***ent produced by the Flat File Assembler.
However this was not the case.
For example
The following record in the input do***ent
ZZX+1468+3+9314916133716:EN'
ZZY+F++:::GLZ K?'BURRA JEWEL STAG:0x920x0'
ZZZ+AAA:134.56000:CA:INV:1:EA'
Was created as the following by the Flat File Assembler
ZZX+1468+3+9314916133716:EN'
ZZY+F++:::GLZ K'BURRA JEWEL STAG:0x920x0'
ZZZ+AAA:134.56000:CA:INV:1:EA'
So in summary I think there are two problems that I’m having.
The first is that escape characters are being added to the output produced
by the Flat File Assembler to escape characters that are the same as
characters used in the Record Tag Identifiers.
The second problem is an escape character that is recognised and removed
by
the Flat File Disassembler is not added to the output file that is
produced
by the Flat File Assembler.
Has anybody had a similar experience and if so were you able to resolve
the
issues?
I am using BizTalk 2004 and have tried both sp1 and sp2. Both give the
same
results.
Below is the original schema that I created which is causing the problem.
Below that I have included some sample records the can be used to show the
problem.
Any help you could give me would be greatly appreciated
Kind Regards
Ryan Moore
Schema Used
----------------
<?xml version="1.0" encoding="utf-16"?>
<xs:schema
xmlns="http://TestEscapeCharacters.Schema.SampleSchemaNonUnique"
xmlns:b="http://schemas.microsoft.com/BizTalk/2003"
targetNamespace="http://TestEscapeCharacters.Schema.SampleSchemaNonUnique"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<b:schemaInfo count_positions_by_byte="false"
parser_optimization="complexity" lookahead_depth="6"
suppress_empty_nodes="false" generate_empty_nodes="true"
allow_early_termination="false" standard="Flat File" root_reference="Root"
codepage="20127" escape_char_type="char" default_escape_char="?" />
<schemaEditorExtension:schemaInfo namespaceAlias="b"
extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension"
standardName="Flat File"
xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions"
/>
</xs:appinfo>
</xs:annotation>
<xs:element name="Root">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited"
preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true"
sequence_number="1" child_delimiter_type="hex" child_delimiter="0x27 0x0D
0x0A" child_order="postfix" escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element minOccurs="1" maxOccurs="99999" name="LINLoop">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="1" structure="delimited"
preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true"
child_delimiter_type="hex" child_delimiter="0x27 0x0D 0x0A"
escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element minOccurs="1" maxOccurs="1" name="LIN">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="1"
structure="delimited"
preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true"
tag_name="LIN" child_delimiter_type="char" child_delimiter="+"
escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element minOccurs="0" maxOccurs="1" name="LIN1082"
type="xs:int">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1"
name="LIN1229">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="1" />
<xs:enumeration value="2" />
<xs:enumeration value="3" />
<xs:enumeration value="1E" />
<xs:enumeration value="38" />
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1"
name="LINC212">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="3"
structure="delimited" preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true" child_delimiter_type="char"
child_delimiter=":" escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:attribute name="LIN7140" use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="35" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="LIN7143" use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="EN" />
<xs:enumeration value="UP" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="LIN1131" type="char3"
use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="3"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name="LIN3055" type="char3"
use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="4"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1"
name="LINC829">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="4"
structure="delimited" preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true" child_delimiter_type="char"
child_delimiter=":" escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:attribute name="LIN5495" type="char3"
use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name="LIN1082" use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:int" />
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1" name="LIN1222"
type="xs:int">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="5"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1" name="LIN7083"
type="char3">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="6"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element minOccurs="1" maxOccurs="999" name="IMD">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="2"
structure="delimited"
preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true"
tag_name="IMD" child_delimiter_type="char" child_delimiter="+"
escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element minOccurs="1" maxOccurs="1"
name="IMD7077">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="A" />
<xs:enumeration value="C" />
<xs:enumeration value="D" />
<xs:enumeration value="E" />
<xs:enumeration value="F" />
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1"
name="IMD7081">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:maxLength value="3" />
<xs:enumeration value="4" />
<xs:enumeration value="27" />
<xs:enumeration value="35" />
<xs:enumeration value="38" />
<xs:enumeration value="44" />
<xs:enumeration value="54" />
<xs:enumeration value="79" />
<xs:enumeration value="98" />
<xs:enumeration value="126" />
<xs:enumeration value="132" />
<xs:enumeration value="BRN" />
<xs:enumeration value="DSC" />
<xs:enumeration value="F01" />
<xs:enumeration value="F02" />
<xs:enumeration value="F03" />
<xs:enumeration value="F04" />
<xs:enumeration value="F05" />
<xs:enumeration value="F06" />
<xs:enumeration value="F07" />
<xs:enumeration value="F08" />
<xs:enumeration value="F09" />
<xs:enumeration value="F10" />
<xs:enumeration value="IM" />
<xs:enumeration value="STE" />
<xs:enumeration value="TPE" />
<xs:enumeration value="" />
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1"
name="IMDC273">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="3"
structure="delimited" preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true" child_delimiter_type="char"
child_delimiter=":" escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:attribute name="IMD7009" use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="17" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="IMD1131" type="char3"
use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name="IMD3055" use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="3"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="9" />
<xs:enumeration value="60" />
<xs:enumeration value="90" />
<xs:enumeration value="" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="IMD7008_1">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left"
sequence_number="4" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="35" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="IMD7008_2" use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left"
sequence_number="5" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:maxLength value="35" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="IMD3453" type="char3"
use="optional">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="6"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="1" name="IMD7383"
type="char3">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="4"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="PRI">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited"
preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true"
tag_name="PRI" child_delimiter_type="char" child_delimiter="+"
sequence_number="3" escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="PRIC509">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="1"
structure="delimited" preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="true" child_delimiter_type="char"
child_delimiter=":" escape_char_type="default" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:attribute name="PRI5125" use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="AAA" />
<xs:enumeration value="AAB" />
<xs:enumeration value="INF" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="PRI5118" type="xs:float"
use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name="PRI5375" use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="3"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="CA" />
<xs:enumeration value="RN" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="PRI5387" use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="4"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="INV" />
<xs:enumeration value="LIU" />
<xs:enumeration value="PRP" />
<xs:enumeration value="TRP" />
<xs:enumeration value="SRP" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="PRI5284" type="xs:float"
use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="5"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name="PRI6411" use="required">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="6"
justification="left" />
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="char3">
<xs:enumeration value="EA" />
<xs:enumeration value="GRM" />
<xs:enumeration value="KGM" />
<xs:enumeration value="LTR" />
<xs:enumeration value="MTK" />
<xs:enumeration value="MTQ" />
<xs:enumeration value="MTR" />
<xs:enumeration value="PCE" />
<xs:enumeration value="TNE" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" name="PRI5213" type="char3">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="2"
justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:simpleType name="char3">
<xs:restriction base="xs:string">
<xs:maxLength value="3" />
</xs:restriction>
</xs:simpleType>
</xs:schema>
Sample Data
-----------------
LIN+0+3+9314916002470:EN'
IMD+F++:::LOUVRE DOOR:2040x720x33'
PRI+AAA:73.08000:CA:INV:1:EA'
LIN+1+3+9314916002999:EN'
IMD+F++:::LOUVRE SCREEN:2040x360x29'
PRI+AAA:38.86000:CA:INV:1:EA'
LIN+2+3+9314916206533:EN'
IMD+F++:::MERANTI HNG?+STRIKER S/SET:2100x112x19'
PRI+AAA:19.93000:CA:INV:1:EA'
LIN+3+3+9314916087262:EN'
IMD+F++:::PRM MDF REGENCY ARC SET DOUBLE:0x42x12'
PRI+AAA:4.22000:CA:INV:1:EA'
LIN+4+3+9314916084889:EN'
IMD+F++:::GLZ K?'BURRA JEWEL SCSL 4G:0x0x0'
PRI+AAA:70.76000:CA:INV:1:EA'
LIN+5+3+9314916062979:EN'
IMD+F++:::PRM STURT INTERNAL FEATURE:2340x870x35'
PRI+AAA:129.92000:CA:INV:1:EA'
LIN+6+3+9314916206526:EN'
IMD+F++:::MERANTI HNG?+STRIKER S/SET:2100x98x19'
PRI+AAA:19.93000:CA:INV:1:EA'
LIN+1055+3+9314916035683:EN'
IMD+F++:::GLZ LATTICE SPRING BARWON:0x0x0'
PRI+AAA:130.50000:CA:INV:1:EA'
LIN+640+3+9314916075993:EN'
IMD+F++:::REFLECTIONS DOOR ARCH TOP:2040x820x35'
PRI+AAA:196.04000:CA:INV:1:EA'
LIN+640+3+9314916075993:EN'
IMD+F++:::REFLECTIONS DOOR ARCH TOI:2040x820x35'
PRI+AAA:196.04000:CA:INV:1:EA'
LIN+640+3+9314916075993:EN'
IMD+F++:::REFLECTIONS DOOR ARCH TOL:2040x820x35'
PRI+AAA:196.04000:CA:INV:1:EA'
