Delphi Out of Memory IDhttp GET

Hey,

This runs fine until the exception handler outputs "Out of memory" then all threads start saying the same.. Is there a memory leak I'm missing in here? Please help

function TBaseThread.doGet(url: string; useragent: string; UseProxy:Integer) : string;
var
http:TIDHttp;
html:String;
cookiestr:string;
code:integer;
myssl: TIdSSLIOHandlerSocketOpenSSL;
mysocks:TIdSocksInfo;
i:integer;
boolretry:boolean;
retries:integer;
sl:TStringList;
label
retry;
begin
repeat
sleep(100);
application.processmessages;
until paused[myindex]=false;

Result := '';
http := tIdHttp.Create(nil);


//setup
retries := 1;
HTTP.Request.ContentType:='application/x-www-form-urlencoded';
HTTP.Request.Connection:='Keep-Alive';
http.Request.Method := Id_HTTPMethodGet;
HTTP.Request.UserAgent := useragent;
HTTP.HandleRedirects := True;
HTTP.AllowCookies := True;
HTTP.CookieManager := TheCookie;
//http.Port := 80;
//http.Host := url;
http.ProtocolVersion := pv1_1;

mysocks := TIdSocksInfo.Create;
myssl := TIdSSLIOHandlerSocketOpenSSL.Create(http);
myssl.SSLOptions.Method := sslvTLSv1;
http.IOHandler := myssl;

//load cookie
try

for i := 0 to Cookie.Count-1 do begin
cookiestr := cookiestr+ copy(Cookie.Strings[i],1,pos(';',Cookie.Strings[i]))+ ' ';
end;

HTTP.Request.CustomHeaders.add('Cookie: '+cookiestr);

except
addlog('Error Loading Cookie in GET');
end;
boolretry := false;

//Set Time Outs
http.ReadTimeout := ReadTimeout;
http.ConnectTimeout := ConnectTimeout;

//setup proxy
    if UseProxy = 1 then
      begin
        if(ptype = 'socks4')then
          begin
            if (pUser <> '') and (pPass <> '') then
              begin
           // http.Socket.SocksInfo.Authentication := saUsernamePassword;
            mysocks.Host := pHost;
            mysocks.Port := Strtoint64(pPort);
            mysocks.Username := pUser;
            mysocks.Password := pPass;
            mysocks.Version := svSocks5;
            mysocks.authentication := saUsernamePassword;
            http.Socket.TransparentProxy := mysocks;
            end;//if username and pass
            mysocks.Host := pHost;
            mysocks.Port := Strtoint64(pPort);
            mysocks.Version := svSocks4;
            mysocks.authentication := saNoAuthentication;
            http.Socket.TransparentProxy := mysocks;
          end else begin
            if (pUser <> '') and (pPass <> '') then
              begin
                http.ProxyParams.BasicAuthentication := true;
                http.ProxyParams.ProxyServer := pHost;
                http.ProxyParams.ProxyPort := Strtoint64(pPort);
                http.ProxyParams.ProxyUsername := pUser;
                http.ProxyParams.ProxyPassword := pPass;
              end else begin
                http.ProxyParams.ProxyServer := pHost;
                http.ProxyParams.ProxyPort := Strtoint64(pPort);
                http.ProxyParams.ProxyUsername := '';
                http.ProxyParams.ProxyPassword := '';
              end;
          end;
      end;


retry:
try
//addlog('GET Sent');
Result := http.Get(url);
//addlog('GET Received');
except
 on E: Exception do begin
code := http.ResponseCode;
Addlog(e.Message);
boolretry := false;
if(retries < retryattempts) then begin
addlog('Retrying...GET');
 inc(retries);
 sleep(1000);
 http.Disconnect(false);
 boolretry := true;
end else begin
//return skip
result := 'skip';
end;
end;
end;

//retry boolean
if(boolretry) then begin
boolretry := false;
goto retry;
end;



//write debug
  if(debugmode) then
    begin
      if(not(directoryexists('debug')))then
        begin
          createdir('debug');
        end;
      sl := tstringlist.create;
      sl.Text := result;
      sl.Add(URL);
      sl.SaveToFile('debug/GET_'+email+'_Action_'+inttostr(debugpos)+'_.html');
      sl.free;
      inc(debugpos);
    end;

//TheCookie.Free;
myssl.Free;
mysocks.free;
http.free;
end;

Open in new window

nickdelphi777Asked:
Who is Participating?
 
jimyXCommented:
> Seems like the doget function gets a VERY LARGE page

If you have/expect huge data, reduce the number of running threads.

Did you inspect which URL and the size of data you are accumulating from doget function?

You should inspect the data and log some useful details to know your next course of action, like which URLs and what sort of data that causes this issue. That's the only way to handle such problems.

Since your code works without issues sometimes, it means it's the data and the way of handling it is getting you to that error. If it does not go right, go left, if you know what I mean.
0
 
jimyXCommented:
Man, you got to use Try-Finally blocks, that is critical in your situation.
Try
  Comp.Create;
  Comp.use;
Finally
  Comp.Free;
end;

Open in new window


Also, AFAIK idCookiemanager is not thread safe, i.e when used in threads you might get troubles when two threads need to use it at the same time.

Last thing, no need to free the myssl. Every component that was created under owner, the owner takes care of it when this owner gets freed itself.

line 39:
myssl := TIdSSLIOHandlerSocketOpenSSL.Create(http); //http is the owner

line 147:
myssl.Free;  // let the owner takes care of this
0
 
nickdelphi777Author Commented:
I added your suggestions, any other idea why this could be happening? Also it only happens SOME TIMES.. weird
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
jimyXCommented:
> Also it only happens SOME TIMES.. weird
Welcome to multi-threading.

Since "out of memory" error occurs when memory is depleted. There could be some unexpected loops that consume the resources. Even, there could be nothing wrong, just too much data to handle.
I can not tell by looking at that code, nothing indicates what might go wrong. All I can say is log every byte and bit.
Perhaps you need to log more details throughout the execution, what you send and what you receive, everything, to find out what has happened/changed when you got the error, and at which code block.
0
 
Sinisa VukCommented:
besides all other said:
- line
application.processmessages

Open in new window

is quite wrong in thread (not needed at all)
- line:
myssl := TIdSSLIOHandlerSocketOpenSSL.Create(http)

Open in new window

- should be:
myssl := TIdSSLIOHandlerSocketOpenSSL.Create(nil);

Open in new window


more similar questions on EE1 or EE2 as an example of using threads with Indy...
0
 
Geert GOracle dbaCommented:
actually try finally is like this:

Comp.Create;
Try
    Comp.use;
Finally
  Comp.Free;
end;

Open in new window



 and ... you are still using spaghetti coding ...
label retry;
...
goto retry;
0
 
nickdelphi777Author Commented:
I will implement all these suggestions.. it keeps happening and it also says Invalid Handle (6) lol which is pretty much the same thing i guess..
0
 
nickdelphi777Author Commented:
Thread Error: The handle is invalid (6)   any ideas guys? Happens randomly
0
 
nickdelphi777Author Commented:
Thread Error: The handle is invalid (6) happens in the exception code in the doGet function above.. :(
0
 
nickdelphi777Author Commented:
I captured the line it is saying out of memory on.. Seems like the doget function gets a VERY LARGE page or something says out of memory in exception then my program threw another error on the Copy() in this function

function TBaseThread.findvalue2(value:string;html:string;send:string):string;
var
pos1,pos2:integer;
ohtml:String;
begin
pos1 := pos(value,html);
if pos1 > 0 then begin
ohtml := Copy(html,pos1+length(value),length(html));
end;

pos2 := pos(send,ohtml);
if pos2 > 0 then begin
result := copy(ohtml,1,pos2-1);
end else begin
result := '0';
end;

ohtml := '';
html :='';

end;

Open in new window


I'm lost.. lol
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.