SimpleXML_Load_String fails with special characters

There is no ASCII 133 - ASCII is only 7 bits.
In utf-8, the codepoint 133 (U+0085) should be encoded as two octets: 0xc2 0x85
COuld it be that you have only one octet 0x85, hence invalid utf-8?

Yes the field data is enclosed within CDATA tags.

I understood that ASCII 133 was in the extended characterset. The character is listed in the third table down at http://www.idevelopment.info/data/Programming/ascii_table/PROGRAMMING_ascii_table.shtml under DEC value 133. I'm afraid I don't understand the technicalities or significance of the number of octets. It may still be invalid utf-8, but the php function utf8_compliant seems to check it as ok.

It still looks like the input string is rather iso8859-1 than utf-8 (and I don't find a function utf8_compliant at www.php.net).
GIve
SimpleXML_Load_String( utf8_encode($data), ...)
a try.

Ah sorry, utf8_compliant is a function I picked up off the net as shown below.

Using utf8_encode($data) made no difference.

An example of data from the xml file that is causing the problem is shown below the function in the code box below; the data has been reduced to only include the sentence with the offending character, which appears as a black rectangle in the Textpad editor, but shows as 3 dots in the code below.

	// reference http://www.phpwact.org/php/i18n/charsets
	function utf8_compliant($str) {
		if ( strlen($str) == 0 ) {
			return TRUE;
		}
		// If even just the first character can be matched, when the /u
		// modifier is used, then it's valid UTF-8. If the UTF-8 is somehow
		// invalid, nothing at all will match, even if the string contains
		// some valid sequences
		return (preg_match('/^.{1}/us',$str,$ar) == 1);
	}
 
 
    <Full_Desc>
      <en><![CDATA[Modern living located in an undiscovered paradise, this is quickly becoming the perfect destination for anyone wanting more&for less.]]></en>
    </Full_Desc>

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:

OK, the utf8_compliant test does what it is supposed to do and as much as can be expected without too much computational overhead.
Hence in your case the code point 133 is correctly encoded as two octets (0xc2 0x85 as I mentioned above) and not just as a single byte character.

However, I do not see 3 dots in your post and that made me check with the unicode charts:
Codepoint 133 or U+0085 is not a glyph but a control (NEXT LINE)
Codepoint 147 or U+0093 is not a glyph but a control (SET TRANSMIT STATE)
I bet these control codes are invalid in XML.

The correct code point for three dots would be U+2026 (HORIZONTAL ELLIPSIS) and various double quotes can be found at U+201C - U+201F

I suspect that the original data was
- produced as windows1250 (or another 125x code page),
- then wrongly interpreted as iso8859-1,
- then encoded as utf-8.

You should replace all these invalid characters with their corresponding intended characters before invoking simplexml_load_string

The offending character showed 3 dots when I pasted it into the code textarea, but I see it now shows as an ampersand.

>You should replace all these invalid characters with their corresponding intended characters before invoking simplexml_load_string
How can I do that (or remove them) if I don't know what they might be. Can I do a regex to remove all characters outside a valid range?

Is DOMDocument better at dealing with such characters?

I suggest you have a look at the contribution by user squeegee on http://de2.php.net/manual/de/function.utf8-encode.php

The data file is 0.5GB in size so I think any parsing function would take too long. I never had an issue before with DOMDocument.

It might work, but without correcting the illegal characters you will at least have unintended / garbled content

I've just tested using DOMDocument and it also errors out with:
DOMDocument::loadXML() [domdocument.loadxml]: Input is not proper UTF-8, indicate encoding ! Bytes: 0x85 0x66 0x6F 0x72 in Entity, line: 109

I tried cleaning up the xml using the function fix_latin() but it fell over at:
if(1==preg_match($nibble_good_chars,$input,$match)){
with the following error:
Warning: preg_match() [function.preg-match]: Empty regular expression

I'm starting to get some where by finding the individual characters that are causing errors and doing a search and replace. I replace a character and test for the next one. But I'm stuck on a character that seems to be ASCII 150 but replacing it does not solve the error, only if I manually delete it will the error go away.

$search = array(chr(150), chr(133), chr(147), chr(148), chr(149), chr(146), chr(163));
$xml_encoded = str_replace($search,'',$xml_encoded);
                                              

1:
2:

The error returned by libxml_get_errors give me the line number. If I extracted the line number from the error message is there anyway I can remove the field value on that line (replace text between tags on that line with nothing) from the string without having to write to disk.