my "ÿþ" mystery

[pryjda]

I am having difficulties explaining this one.
my hopes are that some one in this forum has seen this before. I appreciate any guidance offered.

I have created a script that creates .txt files from my FM database. each record is named after the record number xxxxxxx.txt . A little html scripting is then added  for my import to my ecommerce web site. the text files look great but when the *.txt files are loaded on to the web, the mysterious "ÿþ" characters appear before my text.
I have been trouble shooting this problem now for several weeks. with no success. below is an example of my FM created text file: (this is exactly what FM makes for me)

##################################################

<span class=item>TV/Radio/Lantern with Weather Alert</span>

<span class=subhead></span>

<span class=bodycopy></span>
<span class=featureshead>F E A T U R E S</span>

<span Class=features>• 5" solid state UHF/VHF black and white television, Powerful halogen spotlight and AM/FM/weather band radio with Automatic Storm Alert warning alarm
• 7 channel NOAA weather frequencies warn of dangerous weather conditions and emergency information
• 120 volt AC and 12 volt DC power adapters included
• Built in hand crank provides back-up power for all radio functions and LED area light</span>

#################################################

Now if i copy and paste this information into a new *.txt file or create a new text file from scratch, loading to the web is no problem what so ever. I ma only having this problem with the FM created files. ?

Has any one seen this type of behavior before?
could i be creating my exported files wrong.

Please put your heads together and let me know if you have any ideas. I don't want to manually create 25,000 files every month.

Thank you all for viewing my post and an extra THANKS to anybody who could help.

[Michael Petrov]

I believe this problem is likely caused by some encoding settings that were incorrectly set or defaults which are incorrect for your system. Would you kindly post an exact binary copy of that txt file (place the file into an archive such as zip to preserve its binary structure when transferring) - then I will take a look at the file using a hex editor to see exactly what those characters are. Are you creating those files from within FileMaker client or are you using Custom Web Publishing?

If you are doing it from FileMaker client than the exact export script step / calculation would also be very useful in this case.

Best Regards,
Michael Petrov

[pryjda]

Thank you for your help Michael.
actually i guess i could trace this back, my field contents is actually created by 2 steps


Here is my my field calculation:

"<span class=item>"  &  description_headline & "</span>
" & "¶" &
"<span class=subhead>" & description_subhead  & "</span>
" & "¶" &
"<span class=bodycopy>"  & description_internet & "</span>
" &

If(IsEmpty(description_features); ""; "<span class=featureshead>F E A T U R E S</span>
" & "¶" & "<span Class=features>" & description_features & "</span>")



Here  is my export script:

 Loop
     Set Varriable [$path; value:"servername/directory/directory/"& ITEM_INFO::ROITNO & ".txt"]
     Export Field Contents [ITEM_INFO::HTML4infor Copy; "$path"]
     Go to Record/Request/Page [Next; Exit after Last]
End Loop

i am afraid i can not upload a ZIP file to the forum
i would be happy to email it to you just let me know

Thanks again

[Michael Petrov]

Well all the steps look fine, quite likely its the Export Field Contents command that is encoding it improperly. You could email the zip file to michael [at] fmwebschool.com or upload it on the test page for our new Custom Web Publishing plug-in Grab-It.

Best Regards,
Michael Petrov

[Michael Petrov]

I have looked at the file in the zip and just as suspected it is a simple case of unicode/UTF-16. I have looked around FileMaker and the best internal solution I found was to specify the "Output file character set" within the "Export Records" script step. The correct character set that your web application expects is the simple ASCII (DOS) encoding. The problem with this script step is that it outputs the whole found set into a single file instead of one record. To overcome this limitation I found a solution of omitting the current record and then showing the omitted record only to isolate the current record in the found set.

Another method is to modify the import script to accept unicode. Assuming your web ecommerce solution uses PHP then it can be accomplished  through the following function that I just wrote:

The encoding is utf16 and the function is making an assumption that all characters are simply ASCII and do not include any non-English language characters.

Code:

function utf16_to_ascii($str) {
   return substr(str_replace("\0",'',$str),2);
}

Within your PHP import script just use utf16_to_ascii on the variable that holds your exported text files.

EDIT:
I also forgot to mention that ÿþ are the hex values 0xFF and 0xFE which is a signature of the UTF-16 file.

[Beverly Voth]

This looks like a PHP question (and answer) in the XML/XSLT forum. Please let me know if you need XML/XSLT answer?

If you are EXPORTING as text (tab-delimited) you may also be getting the ASCII 11 (vertical tab). This is the character that replaces the return-in-field upon export, so that the return in the record/row delimiter. If you export as XML, the character is not converted.

Beverly

[Michael Petrov]

This looks like a PHP question (and answer) in the XML/XSLT forum. Please let me know if you need XML/XSLT answer?

If you are EXPORTING as text (tab-delimited) you may also be getting the ASCII 11 (vertical tab). This is the character that replaces the return-in-field upon export, so that the return in the record/row delimiter. If you export as XML, the character is not converted.

Beverly

Hi Beverly,

I agree that it does sound like a web publishing question, that is why I was careful to include the clause "if your eCommerce solution is PHP based" in the reply since the technique would need to be rewritten for other solutions.

I believe that the XML export would add a bit more data then he was asking for, and since he was using a loop to browse the records - the export is done one record at a time. I do not believe that it would do anything to the newline characters, but would the tab export alter any characters from the fields other than ASCII 11 if only a single record was exported?

Best Regards,
Michael Petrov

[Beverly Voth]

I believe that the XML export would add a bit more data then he was asking for, and since he was using a loop to browse the records - the export is done one record at a time. I do not believe that it would do anything to the newline characters, but would the tab export alter any characters from the fields other than ASCII 11 if only a single record was exported?

Yes, a tab-delimited EXPORT also change tabs to spaces within a field. An XML export (using an XSLT) would render the correct characters and the data needed. Since the original calc contains some things like "<" and ">", I'd couch them in CDATA within the calc. That is if the export is as XML...

OR let the XSLT created the needed formatting (all the "<span class=...").
Beverly