Pirates@Home logo

Pirates@Home

Berkeley Open Infrastructure
BOINC!
for Network Computing
Home Help Status Forums Glossary Account

Couldn't connect to database

log in

Advanced search

Message boards : Help! : Couldn't connect to database

Author Message
Profile Kalessin
Volunteer tester
Avatar
Send message
Joined: 19 Jan 06
International
SETI.Germany
Credit: 44,825.1
RAC: 0.00
Joined: Jan 19, 2006
Verified: Jan 11, 2009
Dubloons: 3
Pieces of Eight: 7
Punishment: Mess Duty
Message 6186 - Posted: 8 Jul 2007 | 22:50:40 UTC

During the last weeks I often received this error. Especially (It seems to me) while there is app. run.(Besides this no particular pattern visible to me. Mostly this occurs for five Minutes but sometimes last for hours). Is this due to the increased attempts to connect or are you trying something.

Me Dragon uses the little fiery Foxy 2004
____________
Dragons can fly because they don't fit into pirate ships

Profile Wormholio
Captain
Avatar
Send message
Joined: 6 Jun 04
United States
Away
Credit: 4,065.6
RAC: 0.00
Joined: Jun 6, 2004
Verified: Mar 13, 2008
Dubloons: 3
Pieces of Eight: 10
Punishment: Aztec curse
Message 6189 - Posted: 9 Jul 2007 | 12:39:12 UTC - in response to Message 6186.

Kalessin wrote:
During the last weeks I often received this error. Especially (It seems to me) while there is app. run.(Besides this no particular pattern visible to me. Mostly this occurs for five Minutes but sometimes last for hours). Is this due to the increased attempts to connect or are you trying something.

Yes, I've seen the same problem, and it's even impacted the I2U2 mock-up site I'm hosting here. Basically the DB server decides to stop working when the system load goes above a certain threshold, and there are semi-periodic spikes in load, which I've been trying to track down. The primary cause just seems to be the load on the database during an app run. But there may be other things that contribute, and I try to fix them when I find them.

Thank you for the report, and any observations which might help track down any more contributing factors.

____________
-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats

Profile Pepo
Chief Petty Officer
Volunteer tester
Avatar
Send message
Joined: 13 Sep 04
Slovakia
TeamVision42
Credit: 928.1
RAC: 0.00
Joined: Sep 13, 2004
Verified: Aug 4, 2009
Dubloons: 3
Pieces of Eight: 5
Punishment: Cat o' Nine Tails
Message 6194 - Posted: 9 Jul 2007 | 22:02:23 UTC
Last modified: 9 Jul 2007 | 22:03:59 UTC

The missing DB caught me today for the first time, but multiple times and it took some minute-two.

I've seen two versions:

Error: 1045
Access denied for user 'apache'@'localhost' (using password: NO)

multiple times, and
Error: 1040
Too many connections

but only once (or twice?).

This gave me the idea to take a look at the Project status. Initially it was a cached data of
17:25:05 up 54 days, 3:37, 4 users, load average: 3.87, 2.87, 2.59
but then later I've got this nice
17:45:07 up 54 days, 3:57, 4 users, load average: 17.63, 9.39, 5.07

____________
Peter .-)

Profile Contact
Volunteer tester
Avatar
Send message
Joined: 29 Aug 04
Canada
BOINC Synergy
Credit: 37,127.0
RAC: 0.00
Joined: Aug 29, 2004
Verified: Nov 10, 2011
Dubloons: 3
Pieces of Eight: 3
Punishment: Misfit
Message 6199 - Posted: 12 Jul 2007 | 1:47:42 UTC

For the past several hours not only have I seen no database errors, but also all pirate pages load noticeably faster.
I'm constantly very impressed by this crew & Captain.
Another well deserved patented sitting ovation!
____________
have fun

Profile Wormholio
Captain
Avatar
Send message
Joined: 6 Jun 04
United States
Away
Credit: 4,065.6
RAC: 0.00
Joined: Jun 6, 2004
Verified: Mar 13, 2008
Dubloons: 3
Pieces of Eight: 10
Punishment: Aztec curse
Message 6201 - Posted: 12 Jul 2007 | 13:02:43 UTC - in response to Message 6199.

Contact wrote:
For the past several hours not only have I seen no database errors, but also all pirate pages load noticeably faster.
I'm constantly very impressed by this crew & Captain.
Another well deserved patented sitting ovation!

I don't know if any credit is deserved for this. During that period I did just about nothing to the server.

I've now ramped up the cushion to get lots of work out there, to try to reproduce the problem (yes, we now want it to happen). I'll then try a new version of the feeder, which might be the culprit.

____________
-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats

Profile Contact
Volunteer tester
Avatar
Send message
Joined: 29 Aug 04
Canada
BOINC Synergy
Credit: 37,127.0
RAC: 0.00
Joined: Aug 29, 2004
Verified: Nov 10, 2011
Dubloons: 3
Pieces of Eight: 3
Punishment: Misfit
Message 6202 - Posted: 12 Jul 2007 | 14:35:50 UTC - in response to Message 6201.

Wormholio wrote:
During that period I did just about nothing to the server.

Hmmmm. Maybe then something has changed in the pipeline between here & there.
Or could it be that someone has stopped running (or modified) a script against the pirate server?

Before yesterday, and for many weeks before that, if I was trying to navigate from a pirate page to another pirate page (even the Glossary) it would take as long as 20 seconds for that page to load.
At some point yesterday all pirate pages were loaded almost instantly. This is still the case now.
I've also not seen any database errors since yesterday. A few days ago I got database error even when trying to enter the Glossary!
I noticed this on two very different isp's that are many miles apart.

Ha! Just got a database error during my 1st attempt to post this, but navigation is still fast.
____________
have fun

Profile Pepo
Chief Petty Officer
Volunteer tester
Avatar
Send message
Joined: 13 Sep 04
Slovakia
TeamVision42
Credit: 928.1
RAC: 0.00
Joined: Sep 13, 2004
Verified: Aug 4, 2009
Dubloons: 3
Pieces of Eight: 5
Punishment: Cat o' Nine Tails
Message 6203 - Posted: 12 Jul 2007 | 15:18:57 UTC - in response to Message 6202.

Contact wrote:
Before yesterday, and for many weeks before that, if I was trying to navigate from a pirate page to another pirate page (even the Glossary) it would take as long as 20 seconds for that page to load.
At some point yesterday all pirate pages were loaded almost instantly. This is still the case now.

As I was reading these lines, I've tried to open the Project Status page, in order to see, whether there is any excessive load. Some 20 minutes later, the page is still not yet opened! One minute later I've tried to open the page again in another window and yet another window with this reply - it took twoooooo long minutes. (I did see some 5-6 pages prior to comming here - it was regularly fast.)

On the Status page, the system loads were around 2-3, seems OK. But I've noticed that the page was much older (maybe more than 20 minutes) than the last 10 minutes (there was smething like "CACHED 10:35 AM EDT on 12 Jul 2007 (updated every 10 minutes)" IIRC, but the page was later refreshed automatically, so I've lost the exact data).

And, BTW, is the e-mail notification out of order?
____________
Peter .-)

Profile Pepo
Chief Petty Officer
Volunteer tester
Avatar
Send message
Joined: 13 Sep 04
Slovakia
TeamVision42
Credit: 928.1
RAC: 0.00
Joined: Sep 13, 2004
Verified: Aug 4, 2009
Dubloons: 3
Pieces of Eight: 5
Punishment: Cat o' Nine Tails
Message 6204 - Posted: 12 Jul 2007 | 15:40:54 UTC - in response to Message 6203.

Pepo wrote:
Some 20 minutes later, the page is still not yet opened! One minute later I've tried to open the page again in another window and yet another window with this reply - it took twoooooo long minutes.

Now it is reasonally fast again, but until 3 minutes ago, everything was slow and I was also getting "CACHED 11:02 AM EDT...", then it got replaced with "CACHED 11:20 AM EDT...". So at least the 10 minutes of caching are not true. (The next update I've seen was 11:35 AM EDT, so possibly 15 minutes?)
____________
Peter .-)

Profile Wormholio
Captain
Avatar
Send message
Joined: 6 Jun 04
United States
Away
Credit: 4,065.6
RAC: 0.00
Joined: Jun 6, 2004
Verified: Mar 13, 2008
Dubloons: 3
Pieces of Eight: 10
Punishment: Aztec curse
Message 6205 - Posted: 12 Jul 2007 | 17:39:23 UTC - in response to Message 6203.

Pepo wrote:

And, BTW, is the e-mail notification out of order?

It was, but that should be fixed now. My ISP changed servers on me and there are still some config issues, but I found a relay I can use (the old pirates server, in fact). The one problem I am aware of is we can't send to CERN right now.

____________
-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats

Profile Wormholio
Captain
Avatar
Send message
Joined: 6 Jun 04
United States
Away
Credit: 4,065.6
RAC: 0.00
Joined: Jun 6, 2004
Verified: Mar 13, 2008
Dubloons: 3
Pieces of Eight: 10
Punishment: Aztec curse
Message 6206 - Posted: 12 Jul 2007 | 17:48:52 UTC - in response to Message 6204.

Pepo wrote:
Pepo wrote:
Some 20 minutes later, the page is still not yet opened! One minute later I've tried to open the page again in another window and yet another window with this reply - it took twoooooo long minutes.

Now it is reasonally fast again, but until 3 minutes ago, everything was slow and I was also getting "CACHED 11:02 AM EDT...", then it got replaced with "CACHED 11:20 AM EDT...". So at least the 10 minutes of caching are not true. (The next update I've seen was 11:35 AM EDT, so possibly 15 minutes?)


Here is how it works. If there is a cached copy less than 10 minutes old, then that is what you get, and it's fast.

If the cached copy is more than 10 minutes old then it's regenerated, and you have to wait for it (and everybody else). It's slow, I know, but I've done what I can to speed it up. It's faster after the database has been preened.

If your browser has the page open then after 20 minutes it will automatically refresh, which may or may not require building the page again, depending upon if someone else has done so in the past 10 minutes.

This last bit is a new feature which I intend to submit to BOINC. The ratio between the inital cache period and the regeneration period is configurable (I've set it to 2.0).


____________
-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats

Profile Contact
Volunteer tester
Avatar
Send message
Joined: 29 Aug 04
Canada
BOINC Synergy
Credit: 37,127.0
RAC: 0.00
Joined: Aug 29, 2004
Verified: Nov 10, 2011
Dubloons: 3
Pieces of Eight: 3
Punishment: Misfit
Message 6207 - Posted: 13 Jul 2007 | 2:42:30 UTC

Arrrr. Blast!
Every time I'm here since my last post trying to do damage I have no problems other than sometimes a slow load of the server status page.
But then I've never noticed such low numbers:
2 users, load average: 0.43, 0.39, 0.44

I'm guessing that many pirates chose this time for a break and are down below swilling grog.
Yo ho Captain, why not let out all stops and and let the work flow at full steam.
I'll go below and grab some grog for us working pirates. I'll be right back.
____________
have fun

Profile [B^S] ShanerX
Volunteer tester
Avatar
Send message
Joined: 23 May 06
United States
BOINC Synergy
Credit: 7,928.3
RAC: 0.00
Joined: May 23, 2006
Verified: Feb 19, 2011
Dubloons: 3
Pieces of Eight: 4
Punishment: Aztec curse
Message 6208 - Posted: 13 Jul 2007 | 2:57:22 UTC
Last modified: 13 Jul 2007 | 3:00:43 UTC

I've noticed the low numbers for the past week also ... I can recall much higher numbers in the past months - just now: 22:20:03 up 2 days, 4:49, 2 users, load average: 0.46, 0.44, 0.39

pages seem to load much faster since server reboot ... I've been checkin' me plunder about 10 times daily since ye spilled yer chest! btw - great idea for page cache capt'n ...

only noticed one db connection error, over past 2 days, on 6 different computers, maybe I gots the lucky dubloons! can you check server load and see if any correlation?!

7/12/2007 12:46:00 PM|Pirates@Home|Sending scheduler request: To fetch work
7/12/2007 12:46:00 PM|Pirates@Home|Requesting 47480 seconds of new work

7/12/2007 12:46:05 PM|Pirates@Home|Scheduler request failed: HTTP internal server error

7/12/2007 12:46:05 PM|Pirates@Home|Deferring communication for 11 min 52 sec
7/12/2007 12:46:05 PM|Pirates@Home|Reason: scheduler request failed
7/12/2007 12:58:02 PM|Pirates@Home|Sending scheduler request: To fetch work
7/12/2007 12:58:02 PM|Pirates@Home|Requesting 47498 seconds of new work
7/12/2007 12:58:07 PM|Pirates@Home|Scheduler RPC succeeded [server version 509]

____________

Profile Contact
Volunteer tester
Avatar
Send message
Joined: 29 Aug 04
Canada
BOINC Synergy
Credit: 37,127.0
RAC: 0.00
Joined: Aug 29, 2004
Verified: Nov 10, 2011
Dubloons: 3
Pieces of Eight: 3
Punishment: Misfit
Message 6213 - Posted: 13 Jul 2007 | 5:13:05 UTC

CACHED 12:53 AM EDT
2 users, load average: 4.46, 1.64, 0.79
No problems that I notice.
____________
have fun

Profile Wormholio
Captain
Avatar
Send message
Joined: 6 Jun 04
United States
Away
Credit: 4,065.6
RAC: 0.00
Joined: Jun 6, 2004
Verified: Mar 13, 2008
Dubloons: 3
Pieces of Eight: 10
Punishment: Aztec curse
Message 6219 - Posted: 13 Jul 2007 | 12:15:20 UTC - in response to Message 6207.



Good idea. If we are going to give it a test, let's give it a real test. I had increased the cushion to 7, but now I've raised it to 17, and we'll go higher if necessary.

The cushion is the number of WU's to have hanging around waiting to be served. I can also limit the number which are served per host, and we can play with that too, so see if we can fully load the cluster. We'll see if this generates more of these high load events, or if the number stays the same. Then we'll swap in the new feeder and see what that does.

Looking at the status page for the system load may show you evidence of this problem, but maybe not. The load is sampled every 5 minutes, while one of these events only lasts a minute or two. Here is a screenshot just after such an event:
____________
-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats

Profile [B^S] ShanerX
Volunteer tester
Avatar
Send message
Joined: 23 May 06
United States
BOINC Synergy
Credit: 7,928.3
RAC: 0.00
Joined: May 23, 2006
Verified: Feb 19, 2011
Dubloons: 3
Pieces of Eight: 4
Punishment: Aztec curse
Message 6221 - Posted: 13 Jul 2007 | 14:50:52 UTC

10:45am EST - 4 users, load average: 1.91, 1.85, 1.41

11 wu's in queue, and I've received and crunched several wu's this morning without issue! Aaaargghh .... keep feeding 'em!!
____________

Profile Kalessin
Volunteer tester
Avatar
Send message
Joined: 19 Jan 06
International
SETI.Germany
Credit: 44,825.1
RAC: 0.00
Joined: Jan 19, 2006
Verified: Jan 11, 2009
Dubloons: 3
Pieces of Eight: 7
Punishment: Mess Duty
Message 6230 - Posted: 13 Jul 2007 | 22:09:57 UTC - in response to Message 6219.

Wormholio wrote:
Good idea. If we are going to give it a test, let's give it a real test. I had increased the cushion to 7, but now I've raised it to 17, and we'll go higher if necessary.


That's bringing in the nosy part of the dragon, again:

How does it come to our particular settings, and did they have changed beside the changes within the last run?
(It was very long and intensive even before these last variations before the crash, wasn't it?)
WUs per host?
WUs in the pillows cushions or wherever?
Time and amount of app.run?

____________
Dragons can fly because they don't fit into pirate ships

Profile [B^S] Gamma^Ray
Volunteer tester
Avatar
Send message
Joined: 23 May 06
United States
BOINC Synergy
Credit: 2,356.2
RAC: 0.00
Joined: May 23, 2006
Verified: Jun 23, 2009
Dubloons: 3
Pieces of Eight: 1
Punishment: Walk Plank
Message 6232 - Posted: 13 Jul 2007 | 22:31:30 UTC

Getting this error right now:

7/13/2007 5:30:15 PM|Pirates@Home|Requesting 17280 seconds of new work
7/13/2007 5:30:20 PM|Pirates@Home|Scheduler request failed: Error 403

G^R
____________

5.10.21/Xp-Pro/6600gt/3800x2

Profile Contact
Volunteer tester
Avatar
Send message
Joined: 29 Aug 04
Canada
BOINC Synergy
Credit: 37,127.0
RAC: 0.00
Joined: Aug 29, 2004
Verified: Nov 10, 2011
Dubloons: 3
Pieces of Eight: 3
Punishment: Misfit
Message 6234 - Posted: 13 Jul 2007 | 22:55:02 UTC - in response to Message 6232.

Gamma^Ray wrote:
Getting this error right now:

7/13/2007 5:30:15 PM|Pirates@Home|Requesting 17280 seconds of new work
7/13/2007 5:30:20 PM|Pirates@Home|Scheduler request failed: Error 403

G^R


From the server status page:
The Pirates@Home scheduler is currently restricted to our local domain for testing.

I think we've done what was asked.
Let's see what happens. What a blast!
____________
have fun

Profile Contact
Volunteer tester
Avatar
Send message
Joined: 29 Aug 04
Canada
BOINC Synergy
Credit: 37,127.0
RAC: 0.00
Joined: Aug 29, 2004
Verified: Nov 10, 2011
Dubloons: 3
Pieces of Eight: 3
Punishment: Misfit
Message 6289 - Posted: 16 Jul 2007 | 16:27:14 UTC

Just had a real hard time trying to load pirate pages.
Did we just have another event?
____________
have fun

Profile Wormholio
Captain
Avatar
Send message
Joined: 6 Jun 04
United States
Away
Credit: 4,065.6
RAC: 0.00
Joined: Jun 6, 2004
Verified: Mar 13, 2008
Dubloons: 3
Pieces of Eight: 10
Punishment: Aztec curse
Message 6290 - Posted: 16 Jul 2007 | 19:20:22 UTC - in response to Message 6289.

Contact wrote:
Just had a real hard time trying to load pirate pages.
Did we just have another event?

We've been having many this afternoon, after I raised the cushion to 20, and load jumped almost that high. I lowered the cushion to 5 to see if it will stabilize. I need the DB server to be stable during the day, but we'll load it up again later to try a few things.

____________
-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats

Profile Pepo
Chief Petty Officer
Volunteer tester
Avatar
Send message
Joined: 13 Sep 04
Slovakia
TeamVision42
Credit: 928.1
RAC: 0.00
Joined: Sep 13, 2004
Verified: Aug 4, 2009
Dubloons: 3
Pieces of Eight: 5
Punishment: Cat o' Nine Tails
Message 6291 - Posted: 16 Jul 2007 | 23:52:20 UTC

It is happening now again. Either "Error: 1045 - Access denied for user 'apache'@'localhost' (using password: NO)" (mostly) or "Error: 1040 - Too many connections".

Cached status is now more than 30 minutes old, showing 0 ready results and minimal load averages (0.48, 0.47, 0.61).
Database server status was:
Querys/sec (avg): 17.077
Open tables: 64
Threads: 39

P.S.: Let's see how many (unsuccessful) copies of this message will appear in the thread!
____________
Peter .-)

Profile Pepo
Chief Petty Officer
Volunteer tester
Avatar
Send message
Joined: 13 Sep 04
Slovakia
TeamVision42
Credit: 928.1
RAC: 0.00
Joined: Sep 13, 2004
Verified: Aug 4, 2009
Dubloons: 3
Pieces of Eight: 5
Punishment: Cat o' Nine Tails
Message 6309 - Posted: 19 Jul 2007 | 0:07:05 UTC

Some hour-two ago the web pages were not very responsive. I've tried to open the Status on two pages, the first one opened a minute later than the second one, but the contents - pretty rapid load fall:

Uptime
CACHED 6:26 PM EDT on 18 Jul 2007 (updated every 10 minutes)
alvarez:
18:20:22 up 5 days, 2:09, 4 users, load average: 25.28, 16.91, 9.53

and
CACHED 6:41 PM EDT on 18 Jul 2007 (updated every 10 minutes
alvarez:
18:40:02 up 5 days, 2:29, 4 users, load average: 0.24, 0.87, 3.37


____________
Peter .-)

Post to thread

Message boards : Help! : Couldn't connect to database

Home Help Status Forums Glossary Account


Return to Pirates@Home main page


Copyright © 2017 Capt. Jack Sparrow