Nodejs not returning the response body as expected

Hi,

I'm trying to get request this url:
http://www.tsetmc.com/tsev2/data/search.aspx?skey=%D9%88%D8%A8%D8%B5%D8%A7%D8%AF%D8%B1

If you open this url with a browser you get this:
وبصادر,بانك صادرات ايران,28320293733348826,19382635978639373,33121247877198378,53111266648787119,1,1;

but in nodejs I get strange characters; I also tried loading it with a buffer with utf8 encoding but no success.
The response body I get is:
ucU����      A
�       �V�� �d���      ���>F�=��d�Vb�,�������y�����o��<��%�9�1�Os�{�O�F��C
        ՅP5µ�ʡdްC�VS�NTo*�� 7�m�-�;
                                  �ב2k�"����7vS��0�p�82�F-�y�T�r��Ǚ1

My code is:
var http = require("http");
var request = http.get("http://www.tsetmc.com/tsev2/data/search.aspx?skey=%D8%AA%DA%A9%D9%85%D8%A8%D8%A7", function(res){
    var data = "";

    res.on('data', function(chunk) {
	data += chunk;
    }).on('end', function() {
	console.log(data);
    });

});

request.on("error", function(error){
    console.error(error.message);
});

Open in new window


Any suggestions?
LVL 15
Hamidreza VakilianSenior iOS/Android DeveloperAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bernard S.CTOCommented:
Characters encoding and rendering can be a nightmare at times, and your problem seems to combine all the possible problème in a single shot:
- right to left writing
- character coding

Knowing how to solve all of this Is what you are looking at, but you might consider a more modest target: make it work for your problem.
- I have stopped a long time ago to try solving the fact that displaying some French-accented characters outputs some garbage, AS LONG as the files they are written info have the correct encoding withe the correct characters.
- I agree that getting apparent garbage on a screen is annoying, alarming and painful. However, you know that the ultimate goal of displaying correctly on a web page does work. If you are captuting your (apparent) garbage info a file then examine the gile, is the redult better-looking?
Hamidreza VakilianSenior iOS/Android DeveloperAuthor Commented:
It was not an encoding issue. The problem is the serve sending the http response content in gzip content-encoding format. By default, http library in nodejs doesn't convert the gzipped body, so I had to pipe it to zlib library and the convert it to text. The code is as:

var zlib = require('zlib');
var http = require("http");
var request = http.get("http://www.tsetmc.com/tsev2/data/search.aspx?skey=%D8%AA%DA%A9%D9%85%D8%A8%D8%A7", function(res){
    var body = "";

    res.on('error', function(err) {
       next(err);
    });

    var output;
    if( res.headers['content-encoding'] == 'gzip' ) {
      var gzip = zlib.createGunzip();
      res.pipe(gzip);
      output = gzip;
    } else {
      output = res;
    }

    output.on('data', function (data) {
       data = data.toString('utf-8');
       body += data;
    });

    output.on('end', function() {
	console.log(body);
    });
});

request.on("error", function(error){
	    console.error(error.message);
});

Open in new window



Administrator can close the question.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Hamidreza VakilianSenior iOS/Android DeveloperAuthor Commented:
It was not an encoding issue. The problem is the server sending the http response content in gzip content-encoding format. By default, http library in nodejs doesn't convert the gzipped body, so I had to pipe it to zlib library and the convert it to text.
Bernard S.CTOCommented:
Thx for this nice solution. Another item on my personal check-list for these encoding issues!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Scripting Languages

From novice to tech pro — start learning today.