?
Solved

HTTP messaging

Posted on 2007-07-27
15
Medium Priority
?
198 Views
Last Modified: 2013-12-25
I want to implement a client application that uses PERL and CURL to send messages via http from one remote server to another database server. THe database will query the database or update the database and send the results back to the perl/http client. perl scripts on the remote server will be running continuously and perform file management on the server based on the XML data coming out from the database.  I am going to format db output as an XML because it is easier to parse by perl on the other server.

I read on another site that http is not robust for this kind of application. Also, there will be issues on how to handle timeouts and errors from the database program.

The problem is that my database server is behind a firewall and can only be accessed via http calls.

Has anyone done a similar application and is it true that the client API will have issues with timeouts, transactions, errors, etc.
0
Comment
Question by:munzer
  • 7
  • 6
  • 2
15 Comments
 
LVL 7

Assisted Solution

by:ravs120499
ravs120499 earned 600 total points
ID: 19586289
Yes and no. Going through a firewall and all the network hops that implies does make it more likely to have network timeout issues than an intranet environment.

However, so long as you are not intending to route a database api call (odbc, jdbc, etc.) through the firewall, this is not much different from standard web based apps - they also use some database for their underlying persistence. So long as you follow the basic "good design" principles for web based apps, you should be fine.

Unless I am missing something from what you have described...

- Ravs
0
 
LVL 39

Expert Comment

by:Adam314
ID: 19592513
Are you planning on using HTTP to talk directly to the database, or will you be talking to a webserver, that is then talking to the database?
0
 

Author Comment

by:munzer
ID: 19597504
well we have two unix remote servers: one has an oracle database and the other has a file system.
the database machine is behind a firewall. The oracle application server (mod_plsql) which allows you to run oracle stored procedures over the web is outside the firewall (3rd machine in front of the database). The perl or filesystem machine will call the oralce app server using http. Then the web server will forward request to oracle gateway which talks to the database via NET*8.

The remote filesystem server basically will send a message (post data as a Query or Update) to the database and then does its file management based on that. For exampl,e if a user flagged a certain book to be deleted in the database,a perl batch job will be sent to the database to perform a query, and the db will send  that book number. perl will delete that book from the filesystem and then send an update message to the database telling it the book has been deleted. The database updates a flag and then send perl on the other server confirmation that "Update OK" or "Update failed'.

This is where I am wondering whether "HTTP" will work or not. The problems that need to be handled are timeouts and errors.
0
Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

 
LVL 7

Expert Comment

by:ravs120499
ID: 19598690
"Oracle Gateway" in your post - is it the same as the web server running mod_plsql?

Basically, if you are trying to tunnel plsql calls through a firewall, it is not a standard or easy thing to do. I know of apps that have done it, but they are mainly legacy apps and they took a long time and lots of tweaking to reach stability. Can you not use HTTP through the firewall and run an HTTP - SQL gateway *inside* your corporate intranet?

- Ravs
0
 

Author Comment

by:munzer
ID: 19600659
yes, mod_plsql is the oracle gateway and it is extension to the apache web server.

I do not understand what you mean by tunneling plsql calls through a firewall, but all the web applications we have run fine using this configuration. You are simply submitting an http request to the web server outside the firewall who forwards it to the mod_plsql (also outside) which connects to the database machine inside the firewall, get the data and sends it back to the client.

I do not want to have mod_plsql inside the firewall. That way you need VPN to access it. We want that to be public web server.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 19600733
As long as the network between your three servers is reliable, you shouldn't have timeout problems.
0
 

Author Comment

by:munzer
ID: 19721523
I think the worrisome thing of using http for messaging can be described here. Any commnets on this?

  -  Guranteeed Execution - The whole idea behind having a computer program instead of doing a job
by hand is that a program can run unattended.  HTTP is not a reliaable protocol in that it does not
gurantee delivery or a response. If you need this kind of guarnatee, either write the code to retry
requests yourself or arrange to send your requests through an intermediary whol will perform these
retries for you. Newer versions of the specification allow for using protocols such as JMS to
resolve this issue, but the majority of web services out there still utilize HTTP which does not.
  -  Performance - will vary from time to time.  Performance critical systems are not suited to
become Web Services
  - Availability - Internet is not 100% available
0
 
LVL 39

Accepted Solution

by:
Adam314 earned 900 total points
ID: 19730538
The first comment:  what this is saying is that HTTP doesn't have a retry until success built-in.  This isn't a problem.  You can send a request - if you don't get a response, send it again.  If you are going through the general internet, then if any of the connections between your computer and the server are down, you won't be able to connect.  If you are just going between a few computers on your local network, then as long as your local network (and the server is running)is up, you should be able to connect.

The second comment:  If your webserver is open to the public, and might get very busy - this can slow it down.  If your app is the only thing connecting to this webserver, that shouldn't be a problem.

The third comment: This goes along with the first - if any connection between the computer making the request and the server is down, you won't be able to connect.  This has nothing to do with HTTP though - any protocol going through the internet would have this liability.
0
 

Author Comment

by:munzer
ID: 19732258
well if you send a request, the request may get executed (update a record) but you did not get a response. So now, you will try to send a duplicate request again (another update). It is like mailing an same enveloper two or 3 times until you the other party tells I received it.  Then the other party has no idea that you did get his response back so he wants to hear back from you that you got his reposnse. This is the issue with HTTP. PLus between the two times you send your update a user may have done some updates, so you run into concurrency issues.

The general comment about interet aplies to any protocol. This is why high availability systems do not use internet or even highly secure systems. They will have their own network.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 19734345
You can use transactions to account for any updates that might occur from your initial request to any updates.  If  you setup your application correctly, you can also account for the client sending multiple requests but not getting a response.

Yes, HTTP is not the most robust, but if that is what you have to work with, you can make it work quite well.
0
 

Author Comment

by:munzer
ID: 19735351
what do you mean by using transactions to account for any updates? Can you give a real example?

You can use the mail example? You mail me a letter, I receive it but you did not receive a confirmation from me. The end result should be that both of us confirm that I received the letter, did read it (transaction) and then send you a response and made sure you received a response. Then that would be a successful transaction This can be non-ending, because if you want to confirm that I received the response then I have to send you something back and we keep going in a loop.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 19738152
Transactions are supported by many databases.  It basically makes sure that a set of commands happen together, without changes occuring in between.  Here is a page if you are using MySQL:
http://dev.mysql.com/books/mysqlpress/mysql-tutorial/ch10.html
0
 

Author Comment

by:munzer
ID: 19741837
Thanks for the link. I will read it tonight, even though I do understand the concept of transactions in oracle. You have several tasks and you either commit all of them or not to make sure all of them happen.

However in this case where you use HTTP. Client sends a request to perform a transaction. DB does the transaction and wants to send a confirmation to client. Server goes down or internet is not working. Client did not get an http message about transaction. You can not rollback the DB transaction at this step? Client will try to send same request again even though it was performed.

unless you do some strange setup where you apply updates to a staging table and then when client says I got the confirmation, you copy that data from staging table to actual table. But this design is dangerous too.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 19742028
If your client can make a query to determine if a change was made, it would make that check before re-attempting to do the entire transaction.

Eg:
start transaction
.... do stuff ....
commit     #client never get a response from this
do query to check if any piece of "do stuff" was completed (eg: last update field or something)
If any portion of it completed, you can safely assume (if transactions are supported) that the entire transaction completed.  Otherwise, you redo the entire thing.

0
 

Author Comment

by:munzer
ID: 19743102
yes it requires considerable coding. oracle recommends using web services since it provides a lot of the code for that. i just do not know how web services will make it easier. it also runs on http.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction HyperText Transfer Protocol (http://www.ietf.org/rfc/rfc2616.txt) or "HTTP" is the underpinning of internet communication.  As a teacher of web development I have heard many questions, mostly from my younger students who have come to t…
The Windows functions GetTickCount and timeGetTime retrieve the number of milliseconds since the system was started. However, the value is stored in a DWORD, which means that it wraps around to zero every 49.7 days. This article shows how to solve t…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Six Sigma Control Plans
Suggested Courses
Course of the Month13 days, 9 hours left to enroll

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question