Software Crashes once a day

unnameduser

I'm having a wee bit of difficulty and perhaps someone can help.

Big breath ...

A small network I am dealing with has 1 server, Dell PE2800 (SBS 2003 SP3). It is running AD, and acting as a file server. It is also running exchange for emails. There are 4 DELL XP desktop computers that are connecting to the domain via 1 Gb NICs.

Each computer runs NOD32 antivirus.

The network is setup as follows, Each computer + server + eircom netopia BB router connect to 1 Gb switch via patch panel.

Server has static address 192.168.1.1 and all other computers receive dynamic IP's. DHCP and DNS is setup on the server (rather than router).

The server runs POS software that is also installed on client machines. The client machines have mapped network drives (m:) to the server where the data is stored. We have logon scripts that ensure that the M: drive is correctly mapped.

My problem is that the network crashes around the same time every day - between 4 - 6ish. The POS application seems to crash if it cannot access the network drive. An error provided by the application says "network error" When the crash occurs I check the network drive and ping the server and it does not seem like the server is down.(unless the network was down for a number of seconds and then came back up)

Having viewed logs on server and client I do not see anything to explain what is happening. I have changed Duplex settings settings etc, and have tried static IP's but to no avail. I have tried changing cables, NIC's, switch and BB router all to no avail.

I know that the it is possible that the error is with the POS software and I should talk to the software providers but I need to ensure that the network is 100% correct first. The last person to try solve this issue failed.

(one thing I noticed from server logs is that the exchange server that checks for emails every 15 mins always records errors when downloading messages, however this happens all day long, not just when the server goes down. Also i read somewhere that this can be down to a bug with NOD32 (?) )

I have been trying to figure out what is happening with the server at the same time everyday but cannot figure it out.

Hope someone can shed some light or else back to the drawing board ...

M@21

Is NOD installed onto the server?
If so have you tried disabling NOD from 4pm to rule it out?
Is there a scheduled scan run at this time or is it real-time scanning?

If all desktops show that the POS application hangs at the same time it would suggest the server is definitely the place to focus on instead of the NOD32 on the desktops.

You could also work on one of the desktops without the POS app running, to verify that your network is working okay and the desktop's connection to the server is functional. Run outlook and say a couple of web pages so as to verify connectivity to the server.
If the desktop hangs then it would suggest a network issue, if it does not then the POS app would be the place to focus on.

Hope this helps.

unnameduser

Thanks for your post. Yes NOD is installed on the server. I will try disable it to see if it helps. There is real time scanning so scans are not running around 4. I run these during the night.

The POS would usually crash on all the machines at once. I have worked on the desktops with the POS running and whilst the other machines crashed, the one I am working on didnt.

It does seem to be a POS app issue.

any other suggestions out there?

seamus

Is it possible that NOD runs scheduled updates around those times?. I know some AV softwares will disable all network adapters while they perform updates (to avoid the chance of infection while they're offline).

unnameduser

it was set to run regular updates - ie once an hr by default. I have amended this to one update per day after hours to see if this has an effect.

Thanks.

Any other suggestions?

Is there anything else I should check - apart from NOD32 issues.

unnameduser

still having issues with this server - it is still crashing but now once every 2 days around 4.35.

The program utilises SQL server so I am thinking of scheduling a SQL service restart once every night at midnight when the program is not working.

Any other suggestions?

10-10-20

Leave a ping running on the clients, but pipe the output to a text file along with a time stamp. You'll be able to see whether the network is failing.
I didn't see a description of how you recover from the issue. Do you have to reboot or what?
If you think that it's application/memory leak related, watch the memory usage via task manager.

unnameduser

i have ran ping -t before. Will try output to text file and see how it goes.

To get over the problem I just have to close the program and reopen it. I have checked task manager and memory does not seem higher than usual so a memory leak is unlikely.

Thanks for your suggestions. Cheers.

10-10-20

It sounds like a memory leak within the application to me. Possibly a page-pool issue.
http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx

If it's a turnkey application, you might be able to enable a software watchdog to reset it on a hang.

unnameduser

unnameduser wrote: »

still having issues with this server - it is still crashing but now once every 2 days around 4.35.

The program utilises SQL server so I am thinking of scheduling a SQL service restart once every night at midnight when the program is not working.

Any other suggestions?

I have scheduled this SQL service restart once a night via a batch file and the server has not crashed in 4 days. Fingers crossed this will continue.

If the sql service is to blame can anyone suggest a reason it is failing? I could leave the batch process in place but ideally I'd like to know why it is failing.

Alzar

unnameduser wrote: »

....If the sql service is to blame can anyone suggest a reason it is failing? I could leave the batch process in place but ideally I'd like to know why it is failing.

Really does sound like a memory leak of some sort as 10-10-20 has suggested. Especially as there has been no server crash since you began to restart the SQL Server Service daily.
There are 2 ways to approach this:
1. Leave it as is & schedule the batch file to restart the SQL Server service
or
2. Get down & dirty & troubleshoot the problem & see if you can catch the cause of the crash.
Have you read this page here:
http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx
as posted by 10-10-20? Some good info in it.

Al.

ttm

unnameduser wrote: »

I have checked task manager and memory does not seem higher than usual so a memory leak is unlikely.

Not sure if any of the links cover it but I always check the number of handles in use, the memory useage of an app can stay almost static while the number of handles that are opened increases.

Open up task manger and on from view on the file menu click select colums and add in handles. A bad app can very slowly consume more and more untill the system just locks up. Its tricky to catch the point where the system crashes but over a period of time its easy to spot an app that started off using say 1,500 Handles and a day later is using 4,000.

From my experince and pure predudice (just being honest) I'd be lookng long and hard at the POS app. Not sure how it all works from what you've said so far? Is there a main server POS app that you reboot to get everything running again?

unnameduser

I'll be back working on this in the morning.
As regards getting the program back working again it is simply a matter of restarting the POS program on the client machine.

ttm

Sorry but I still can't understand the setup?

Who or what makes the connection to the SQL server? Is it each individual client PC (POS app) or a server POS app? So is there a server POS app or is it all client based connecting to SQL with a managment app for getting out reports etc?

When you restart the POS app to get it working is it the POS app on just one client if not which one and what roles does it perform.

Reason I'm asking is if SQL is croaking is I'd want to know what was writing data to it or making a queries. If there is a managment app for the POS clients sitting on the server does it create a report at a particular time each night?

So if you can explain the setup a bit more we might be able to point to areas to take a closer look at. btw you can't rule out anything, I really did have one server that crashed every weekend when the cleaner pulled the plug out to use the vacuum cleaner :rolleyes:


Edit> Just re-read the OP and I can see there is POS software on the clients and the server but is it the same software on client and server and if not which one connects make the SQL connection