Missing Characters after Umlaut (ä,ö,ü) in HttpRequest.responseText

hi, so im using an RSS Content feed class ( http://www.tele-pro.co.uk/scripts/rss/rss_content_feed_class.htm ) and knowticed that some RSS feeds gave me a syntax error while validating via the MS phraser.( MSXML2.DOMDocument.loadXML )
after a bit of research i quickly knowticed that the problem was with the umlauts.
Finaly i simply read out the ( http.responseText ) and knowticed that follwing an umlaut the next 6 characters were missing.

thus out of the original
 <title>Kunst: Komprimiertes Glücksgefühl</title>
 <description>

resulted:
 <title>Kunst: Komprimiertes Gl?f?itle>
 <description>

(taken from http://newsfeed.zeit.de/ )

i thus ge the follwing error: "GetRSS: XML error: End tag 'item' does not match the start tag 'title'. Parse Error line 207, character 7" since the title end tag is missing.

so my question: How do i need to adjust my HttpRequest object in order to get a correct responseText?

Skyruner2Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Skyruner2Author Commented:
it is urgent.
fritz_the_blankCommented:
You will have to encode this:

http://www.raok.ee/programming/index.php?id=4

FtB
fritz_the_blankCommented:
Someone here had the same issue with solutions suggested:

http://www.pmachine.com/forums/viewthread/13747/

FtB
Fundamentals of JavaScript

Learn the fundamentals of the popular programming language JavaScript so that you can explore the realm of web development.

Skyruner2Author Commented:
ok, so it is definaly a problem with the encoding.
Either my HttpRequest object OR the XML supplying server has a wrong setting.

Since i can not change the setting (or supplied XML) from the server (I could send the guys an e-mail, but the server/XML source is not under my controll), i will have to work with the setting of the HttpRequest object.

So the problem is reduced to a setting of the HttpRequest object. Well, is it possible to overwrite the char.encoding? if yes, how?

~Sky


 p.s.: i did not find anything after two quick google searches for "overwrite DOM Http Requst encoding" or simmilar terms.
Skyruner2Author Commented:
if you load the XML that gives me the error in Firefox, or the MS XML notepad, it works just fine. if i take the xml as a string recived by my request obj. then i recive an error.

 http://newsfeed.zeit.de/

www.heute.de (look for news feed (alternate tag in html head))
fritz_the_blankCommented:
Okay, I can do this without error:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title> New Document </title>
<%
Function GetHTML(strURL)
      Dim objXMLHTTP, strReturn
      Set objXMLHTTP = Server.CreateObject("MSXML2.ServerXMLHTTP")
      objXMLHTTP.Open "GET", strURL, false
      objXMLHTTP.Send
      strReturn = objXMLHTTP.responseText
      Set objXMLHTTP = Nothing
      GetHTML = strReturn
End Function
%>
</head>

<body>
<%
strFile=gethtml("http://newsfeed.zeit.de/")
response.write strFile
%>
</body>
</html>
fritz_the_blankCommented:
It looks like the encoding for the xml is "iso-8859-1" so I wonder if you can set that on your .asp page as the encoding type or in your XSL file that does the transform?

FtB
Skyruner2Author Commented:
I stripped my code of all possible post data, and here is what i am now using:

'Retrieve response and return HTML response body
Public Function XmlHttp(xAction, data, hdrs)
  Dim HTTP, Raw
  Set Http = CreateObject("MSXML2.ServerXMLHTTP")
  'MSXML2.XMLHTTP
 
   Http.open "GET", xAction, FALSE

  Http.send '(data)
  Raw = "_"
  response.Write(server.HTMLEncode(http.responseText))
  Set Http = Nothing
  XmlHttp = Raw
End Function


still gets me:



 <title>Kunst: Komprimiertes Gl&#65535;f&#65535;itle>  <description>Wie Ulrich und Sylvia Str&#65535; urpl&#65535;ich zu de

thus still an error (see missing </title> tag)

i am not using any XSL file. what do you mean by "set that on your .asp page as the encoding type" ? do you mean what i have suggested before: overwriting the (wrong) encoding setting of the httpRequest object?
Skyruner2Author Commented:
hmm the aboce &+Charackter code appear as missing charachter, and not as (propably changed druing submition of the message) the code seen above.

so i knowtice the feed does have a header wich identifies the encoding, but the http object does not seem to care much for it!


<?xml version="1.0" encoding="ISO-8859-1"?>
Skyruner2Author Commented:
i got a what seems to be a very usefull text at http://www.devguru.com/Technologies/xmldom/quickref/httpRequest_send.html

if i understood correctly,  the data type passed through .send( ) method will determin the encoding of the returned data.

"[the] acceptable input types are BSTR, SAFEARRAY of UI1 (unsigned bytes), IDispatch to an XMLDOM object, and IStream.
[...]
If the input type is an XMLDOM object, the response is encoded according to the encoding attribute on the '<?' xml declaration in the document. If there is no xml declaration, an encoding attribute of UTF-8 is assumed.

If the input type is IStream, the response is sent as is, without additional encoding. The caller must set a Content-Type header with the appropriate content type."

so, looks like im stuck with either of thouse. While looking up "IStream" (wich i still dont know what it is), i found this:

" HttpRequest.responseStream

The responseStream property is read-only and represents the response entity body as an IStream. This stream returns the raw, uncoded bytes as received from the server. So, depending on the server, this may appear as binary-encoded data (UTF-8, UCS-2, UCS-4, shiftJis, etc).
" at http://www.devguru.com/technologies/xmldom/quickref/httprequest_responsestream.html

deighcCommented:
I'm surprised that the RSS feed specifies ISO-8859-1 as the encoding. I would've thought the UFT-8 would be more appropriate (but what do I know....??).

You can override this programmatically in the XML parser.

After you load the XML into your DOM object you can use the createProcessingInstruction method to create a new <?xml?> node with the appropriate encoding type.

Assuming your XML DOM object is called xmlDOM:

' Load the RSS XML first. Then...
' Create the processing instruction node
set pi = xmlDOM.createProcessingInstruction("xml", "version=""1.0"" encoding=""UTF-8""")
' Append the node to the beginning of the document
xmlDOM.insertBefore pi, xmlDOM.childNodes.item(0)

Skyruner2Author Commented:
but the problem is while loading the XML!

the responsetext of the HttpRequest object returns a string encoded in UTF-8, though it should be ISO-8859-1. Thus after evey non UTF-8 Character (ä,ü,ö - for exsample) the follwoing 6 characters are missing.

Would the original <?xml?> tag be overwriten, and the raw xml string would be converted back to normal?


just to clear it up again:

Server supplied XML(ISO-8859-1) ---loaded into---> HttpRequestObject.ResponseText  (encoding ignored and set to UTF-8) ---Loaded into---> DOMDocument(still in wron UTF-8 encoding) -> Hairball (since 6 characters after every äöü are "removed" due to the wrong encoding, and thus vital data is missing (closing tags for exsample - thus the Phraser hairball.))

deighcCommented:
I don't believe the problem is loading the XML. I think the problem is the way the HTTPRequest object is dealing with the XML string itself.

But a question for you: why do you use HTTPRequest then load an XML string into an XML DOM object?

You can use the load method of the XML DOM object and pass it a URL. This has the advantage of automatically validating the XML when it's loaded and, if there are any errors, the load method will return false:

if xmlDOM.load("http://newsfeed.zeit.de/") then
    ' XML loaded OK. So you know for certain that the XML is OK
else
    ' Parse error in XML
end if

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Skyruner2Author Commented:
" HttpRequest.responseText

The responseText property is read-only and represents the response entity body as a string. The XMLHTTP object tries to decode the response into a unicode string, assuming a default encoding of UTF-8. However, it can decode any type of UCS-2 (big or little endian) or UTC-4 encoding as long as the server sends the appropriate unicode byte order mark. It does not process the XML '<?' coding declaration."

ok great, but how do i tell it wich encoding it is?!
Skyruner2Author Commented:
@deighc

good point. i am simply working with the code from  http://www.tele-pro.co.uk/scripts/rss/rss_content_feed_class.htm

could be so it can handle all of the different "versions" of rss, rdf feeds?! - i will try loading it directly into the xmlDOM
Skyruner2Author Commented:
might be because of the way the feed cache works in this think... dang, looks like ill end up re-writing most of this code.
deighcCommented:
I misunderstood what you meant by HTTPRequest. You meant XMLHTTP.

This is an OK way to do things (but it still probably makes sense to load the xml directly into the XML DOM).

But regardless of how you load the XML into your DOM object (load(), loadXML()) ALWAYS check the return value to determine whether or not the XML is well formed.

This will eliminate the possiblity that the RSS feed itself is somehow mis-functioning.
Skyruner2Author Commented:
great, looks like this is working... just to sum up: its actually a work around to getting the right data from tehe XMLHTTP, but it works.

I jsut implemented a quick and dirty solution, and i am not sure if everything works out correctly... if you dont mind i would like to keep this open untill i have fully changed the code (latest tomorow (12th of july) around noon Central European Time)
deighcCommented:
I'm curious to know what you mean by "a work around".

You shouldn't need any quick and dirty fixes. MSXML is designed from the ground up to work correctly with all character sets and encoding types. I've used it many times with German character sets and never had problems.

But hey, if it works use it....
fritz_the_blankCommented:
Sorry, I am back now, but too late I see?

FtB
Skyruner2Author Commented:
Well its a work around since i am no longer using the XMLHTTP. The original question was (or better established to be) how get the ResponseText method of the request onject to return the results in the correct encoding. now i no longer use the XMLHTTP.

as to "Quick and Dirty" fix:

right now the code reads: Res.load (xml_URL)

it used to be: res.loadXML getResults(xml_URL)


now i did not write the code, and the getResults(URL) function did some work with the XML besides just checking if a cached file was avaliable. So i have to see if any changes done in this function are vital for the functioning of the script with other XML sources, and i have to re-implement caching the source for X hours.
since i do not have the time to do that now, id like to delay accepting your answer untill i have done this work either tonight, or tomorow morning, so i can be certain this workaround (as described above) is working, or if i still want to fall back on using the XMLHTTP at wich point id be stuck with my problem again....

dont worry im not trying to delay giving out points, i simply want to be sure the solution works 100%.
deighcCommented:
Take your time, I'm not after points - I'm just curious to know what works and what doesn't.

And thanks for the more detailed explanation of the class. It makes more sense to me now to know that it uses a caching mechanism.

But if, for your purposes, you find that you can only get the class to work by loading the XML directly into a DOM instead of an XMLHTTP object it should still be quite simple to modify the class code to work properly with the cache.

My guess is that the cache stores a copy of the XML as a file somewhere on the filesystem and checks a timestamp.

You can use the Save method of the XML DOM to save the RSS XML to a file so this should  be quite simple to implement.
Skyruner2Author Commented:
right... you can get full info @  http://www.tele-pro.co.uk/scripts/rss/rss_content_feed_class.htm ...

it is easy to do, but im leaving for holidays on thursday, so i have lots of other things to do ;)!
deighcCommented:
I had a quick look at the class.

It's hard to say exactly how many different ways the XMLHTTP functionality is used. If all you want to do is request a URL that returns XML then loading it directly into an XML DOM will work fine.

The only time you have to use XMLHTTP is if you need more control over the HTTP request. XMLHTTP lets you control various timeouts and (more importantly) load data (XML, binary etc) into the request header.

I **think** in your case loading the data directly into an XML DOM should work OK. But I understand why you'd want to thoroughly test this first.
Skyruner2Author Commented:
yeas, i saw that defining headers was possible... all i currently want to do is to load publically provided rss feeds from vareious news sites, wich should not requiere and special header data...
Skyruner2Author Commented:
seems to be working allright, but even when i load from cache it seems a bit slow:

http://the-n-e-x-u-s.net:8080/newsfeeds.asp

I turned the response buffer off, so you can see it build up the RSS feeds
Skyruner2Author Commented:
hmm woops looks like it actually builds much faster with the response buffer on!.. sorry..
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
ASP

From novice to tech pro — start learning today.