Solved

Need to automate activity log export.

Posted on 2006-07-06
19
725 Views
Last Modified: 2013-12-18
Experts,

I have searched the channel for a similar question but did not find one.  I have a requirement to send Notes activity data to an external repository. Not being a Notes person, I have not been able to find much on how to do this.  Would the best solution be to write an agent that is somehow fired by the log and have the script then read and send the last entry of the log.nsf?  Or should I write vb or script using COM to interface with the log.nsf file.  Either way, I am not sure how to do that (or if it even can be done) but I guess I'll figure it out.  I can not just export the log.nsf manually.  This has to be automated.  Any suggestions? Can anyone steer me in the proper direction?

Thanks
0
Comment
Question by:kmcclinton
19 Comments
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17051349
More questions for you:
- we are talking Domino, aren't we, not Notes?
- what Domino release?
- what are the "activity data" you need?
- when do you need it, I mean every hour or day, or on command?

You're correct probably with the agent, but first we have to investigate the whole problem.
0
 

Author Comment

by:kmcclinton
ID: 17051565
<<<< - we are talking Domino, aren't we, not Notes? >>

Yes of course, Domino, but you knew that didnt you.....

<< what Domino release?>>
Not to sure at the moment...I have to go back to the lab to see. I believe it's 6.5 (that may be the client install). I will get back to you.

< what are the "activity data" you need?>

Just about everything. User activity, smtp, etc. I don't need server startup and normal op. activity but if they are in same log, I will just filter.
 
<<- when do you need it, I mean every hour or day, or on command?>>
Realtime or close to it. which is why I thought if I could fire code and send it.....

Thanks for you help.

0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17051734
Realtime? Nothing in Domino is realtime. So you are going to put a considerable load on a system when you start polling the database every minute or so?

What is so important that you want to see it before Domino can even warn you through its Events?

Could SNMP or an SNMP agent be of some help??
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17051761
0
 

Author Comment

by:kmcclinton
ID: 17052173
sjef.

This is a highly secured network.  SNMP will not be running on ANYTHING on this network. Not firewall,  router, server..nothing.  I am looking for a way to get Domino activity for various reasons but SNMP is not a solution that will be approved.

Thanks

0
 

Author Comment

by:kmcclinton
ID: 17052403
When I say accepted, I mean accepted by the "powers that be". Personally, SNMP would be fine but I know I could not get that accepted.

Thanks
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17054218
Yes, I think I know how ro pull this one off. But, the cost of it can be very high. That's up to you and your "powers" to decide.

The general idea you can find here:
    http://www.openntf.org/Projects/pmt.nsf/ProjectLookup/Trigger%20Happy

It would be to immediate answer to you problem, if it were not for the fact that "Trigger Happy never responds to events in the Trigger Happy Config database or the Log.nsf database. This is necessary for normal operation of Trigger Happy." And "Log.nsf events aren't logged either, because they can get into an infinite loop, where a message is logged, which triggers an event, which logss a message, which triggers an event. ad nauseum."

This means that you'd have to create the DLL's yourself. A lot of tricky work.

Maybe some external tool can do the job for you. If the keywords are highly, secure, Domino, I can smell the bucks underneath ;)

Solutions I found are best described and compared here:
    http://www.gsx.net/clients/gsx/gsxhomepage.nsf/Documents/GSX+Monitor+Comparison?OpenDocument
0
 

Author Comment

by:kmcclinton
ID: 17057729
Thanks sjef for all your help. Unfortunately, there is too much code in this solution.  I am able to use a small bit of code to integrate but using too much will send off whistle to the government.  If I used this, it will take another 6 months of evals to get the project through (each and every time it is used). I have to come up with a cleaner way of doing this. I can use COTS solutions but if i have to compile anything, that would hold everything up while every line of code is inspected.  Am I asking for something that is not readily available or done often?

Regards
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17058792
You have some pretty tough constraints there! Phew!

Did you have a look at GSX solution? It may be everything you need. And if you don't say what the external repository is... Maybe GSX have a good solution, off the shelf. Call them?
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 

Author Comment

by:kmcclinton
ID: 17059130
Thanks sjef

Yes, the project is not a run-of-the-mill Domino integration. The requirements are VERY tight.  It's looking more like I have to write something to do this.  Thought  I'd give the Experts a try to see if something less painful could be done (that i could use of course). Considering my Notes App knowledge goes as far as just knowing how to CREATE a database (not designing an app) or replicate my mailbox. This is going to be a challenge that i actually am looking forward to. I wouldnt be me if I shyed from this sort of stuff.  Anyway, if you or anyone else, have a leads feel free to add it here. I won't close this as of yet. Unless you don't think I'll get a hits on it.

Regards,
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17059392
I think you won't get many hits, but one never knows.

Judging from your "specifications", you need the following:
- create a Domino Extension, using C and the C-API
- a spare Domino server to develop and test it on
- a lot of time (3 months??)

At least, that's how I would do it. And that's how GSX did it, most likely.
0
 

Author Comment

by:kmcclinton
ID: 17059950
I think I will use Java and a API.  Do you think I won't get many hits because of the nature of the question, how it was phrased, or just not enough who would be interested in it?
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17060141
There are less than 10 experts who post regularly, less than 25 who post accasionally, and the rest have all gone. The die-hards will read your question. I hope they have different views! It's not the question itself, although the complexity is making it very difficult.

Java? And COM therefore, or CORBA? Which means you will be polling the server for information. How often are you planning to do that?? Bound to fail, I'm afraid.

IMHO, the only feasible solution is a Domino Extension, a DLL that is called by the server itself before and/or after modifications are made to a database. When the DLL is started, several events can be activated by means of callback functions. Realer time you can't get.

I'll ask an EE-colleague to look into this :)
0
 
LVL 63

Expert Comment

by:SysExpert
ID: 17060444
I can not think of any good solution , but here are some ideas.
1) Set up a separate Domino server and replicate the log file to it. Use that server to do all your log processing, rather than your main server. It will not be realtime, but it will probably not cause your main server to grind to a halt either.

2) Look into the Unix/Linux Domino servers. maybe there is a way to get the log.nsf into one of the Unix system logs. there are plenty of tools that can do what you want on Unix.

3) Use C-API and do it yourself.
The bottom line is how fast do you really need the log info ?
The faster you want it, the more it will affect server performance - probably - unless you have serious Hardware overkill in place.

Uou should check the log.nsf  file, it can create thousands of entries per minute, so you have to decide what you really want.

I hope this helps !
0
 

Author Comment

by:kmcclinton
ID: 17060682
<< Java? And COM therefore, or CORBA? Which means you will be polling the server for information. How often are you planning to do that?? Bound to fail, I'm afraid>>>>

wow.... be more optimistic why don't you

Gents and if not both Gents .. I apologize.

Have you ever heard of domingo? I just found this. It's suppose to be a Javascript-like Java API to Notes/Domino.  I see there are some interesting methods in the Javadoc that I will play with. This might be the way to go.  I hate C++ (bad memories) and will try to avoid it.   I think whatever I come up with I will do what you both have said.  I will write the code to extract the data from the database, comparing this with what is in whatever external repository I may use.  I think the replication idea is great (hadn't thought of that) and I think I will incorporate that into the test environment when (notice I said when) I get this firing. Hardware will not be an issue - we will run what we have to. But, I can not miss events so it may be best to replicate on a seprate server (something else I don't know how to do yet) and have the code poll maybe once a minute. I have to think about failure responses but.. getting ahead here...

 I am on vacation next week and am leaving in about 45 mins to get a jump start. So I won't touch much till next week. Boy I sure like not having to worry about work when I leave here. I couldn't work from home even if I wanted to. Nothing is allowed to leave here. So...(click) the brain switch was just turned off. Thanks for all you help and I will take this up again next week.

Regards
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 17061816
> It's suppose to be a Javascript-like Java API to Notes/Domino
Read again: LOTUSscript-like

> replication idea is great
Will be unreal-time: replication is usually done every 60 minutes on a normal server, but it can be set faster. How ofter replication is actually done depends on the server's load. Once a minute might kill the server.

The only way not to follow a train is to be on it.
Read the accepted answer on this page: http:Q_10152910.html "How to avoid logon password?"

Here's a long thread about the Extension Manager's capabilities:
http://www-11.lotus.com/ldd/nd6forum.nsf/DateAllThreadedweb/b4f10c44a3ec72358525719500434802?OpenDocument

Have a nice holiday! Eh, I forgot what you weren't supposed to think about...
0
 
LVL 18

Accepted Solution

by:
marilyng earned 500 total points
ID: 17064681
Replication>>Once a minute might kill the server<< nah, not for a couple of applications.  We had a couple that we replicated every five minutes because otherwise we'd end up with replication conflicts.   Never did every minute :)  It probably wouldn't finish replication.  We had to extend the time to allow replication to complete, too.

The issues with the logs in log.nsf is that they are text embedded events on a single form.  Being this, the field: BODY is what you have to mine, and in order to get any relevant answers from either the misc or mail events.  If you're not interested in grabbing specific mail or events it's easy enough to mail/export all the logs for a particular day via an agent.

So, I'm not sure what you're trying to mine here.



If not for that, using DECS to automatically grab and send to SQL via odbc would be the easiest, but then you'd need SQL/MYSQL/DB2 server to import as BLOB, and you won't have any meaningful results, unless you use a view for the ODBC collection.


There are these tools to search for specific items in the logs:
http://www-10.lotus.com/ldd/sandbox.nsf/ecc552f1ab6e46e4852568a90055c4cd/a9e07c8b5efe913585256a76004c696a?OpenDocument&Highlight=0,log

http://www-10.lotus.com/ldd/sandbox.nsf/ecc552f1ab6e46e4852568a90055c4cd/c93e73615e56607f85256f810054c34a?OpenDocument&Highlight=0,log

Either one can be modified to email the results.

----------------
Ok choices - pushing out to a text file is possible, and I think an INI or configuration call. But hard on system.

I like replicate to a local machine and mine from there.  When I had to mine the log, I mined from a LOCAL machine, this way I didn't use server resources.   GSX does this, and creates it's own reports so you can filter yourself using the product.

All will leave a footprint.  For instance, if I mined the logs to see how many times Joe Smith accessed the server, my "reads" will show up in the log.

the only way you're going to get real time results is if you mine via ODBC to another database, or maybe push the logs out to a text file.
0
 

Author Comment

by:kmcclinton
ID: 17222976
My apologies for not closing and awarding points.  I have spent most of my last few days in the lab working with the Java API and NotesSQL and have not been too my desk very often.   I appreciate the comments made but feel marilyng's were most informative. Regards,,
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

I thought it will be a good idea to make a post as it will help in case someone else faces these issues. I trust this gives an idea how each entry in Notes.ini can mean a lot for the Domino Server to be functioning properly. This article discusses t…
This article covers general Notes 8.5 troubleshooting information including recreating the Notes\Data folder.
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now