<br> replace \n (newline) is not working

Well, the second one works for me.  Please see http://jsfiddle.net/9oc8pjet/
Maybe your str isn't what you think it is?  You may want to post your script here so that someone can find the problem.

If html tags are part of the string, why won't you use str.replace("<br>", "\n"); ?
Both won't work where? In an HTML page or in a text-area? Depends what result you are expecting and where you want to see it.

I am printing in plain HTML table not in text area

By the way, @Jargonics, str.replace("<br>", "\n") only replaces the first occurrence.

function openNewWindow1(str)
{
loadProgressWindow(500, 250,test)

	alert(escapeHtml(str));
	var newMsg = '';
	
	newMsg += '<table border="0" cellpadding="0" cellspacing="0" width="400">\n';
	newMsg += '<tr>\n';
	newMsg += '	</tr>\n';
	newMsg += '<tr>\n';
	newMsg += '		<td>&nbsp;</td>\n';
	newMsg += '	</tr>\n';
	newMsg += '	<tr>\n';
	newMsg += '		<td align="center">\n';
	newMsg += '			<table border="1" cellpadding="2" cellspacing="0" width="90%" align="center">\n';
	newMsg += '				<tr>\n';
	newMsg += '					<td><br>'+escapeHtml(str)+'<br></td>\n';
	newMsg += '				</tr>\n';
	newMsg += '			</table>\n';
	newMsg += '		</td>\n';
	newMsg += '	</tr>\n';
	newMsg += '	<tr>\n';
	newMsg += '		<td></td>\n';
	newMsg += '	</tr>\n';
	newMsg += '<tr>\n';
	newMsg += '		<td>&nbsp;</td>\n';
	newMsg += '	</tr>\n';
	newMsg += '	<tr>\n';
	newMsg += '		<td align="center">\n';
	newMsg += '			<input type="button" name="button" value="Close" onclick="closeNewWindow()"/>\n';
	newMsg += '		</td>\n';
	newMsg += '	</tr>\n';
	newMsg += '</table>\n';
	
}

Select allOpen in new window

function escapeHtml(unsafe) {
	
	if (typeof unsafe  == "undefined"){
		return unsafe;
	}else
		{

    return unsafe
         .replace(/<br>/g,"\n")
         .replace(/&/g, "&amp;")
         .replace(/</g, "&lt;")
         .replace(/>/g, "&gt;")
         .replace(/"/g, "&quot;")
         .replace(/'/g, "&#039;");
       
         
		}
}

Select allOpen in new window

output displyed in <div> tag

http://jsfiddle.net/9oc8pjet/3/

It still works using your functions (with some minor modifications in openNewWindow1 to make it work as you didn't post everything, but didn't change anything in escapeHtml).  Please see http://jsfiddle.net/261kun30/

I still suspect that your str is not <sathish><br>Kumar<br>

@taoyipai thanks. but <br> is replaced with new line but sathish kumar are in same line not  the 2 line

the output should

sathish
kumar

Select allOpen in new window

I still suspect that your str is not <sathish><br>Kumar<br>//Yes correct some something like the same manaer

escapeHTML not allow me to that ??  is there any way to achive this ??

No, it has nothing to do with your escapeHtml function.  It's the W3C standard.  Any number of consecutive whitespace characters (including TAB, newline, space and others) would be displayed as a single space, unless you specifically tell it not to, for example, wrap the text with a pair of <pre></pre> tags.