WCG problems

[Assimilator1]

I've noticed for a couple of days or so that results haven't been uploading. I tried to find out from WCG's website what was going on, and it seems only the front page is working.
Clicking on multiple links there, they all came up with errors, including their forums.

Anyone know what's going on?

>>>>[edit] Their forums are back up now, latest info here (or at the twitter link provided below by Skillz)<<<<

 

[Skillz]

Their RAID array died. They claim they are able to recover all the data by replacing the controller. IT doesn't work on the weekends so it wont be until Monday when it gets resolved. All sent tasks deadlines will be extended so no one loses any credit for returning work when it comes back online.

https://twitter.com/WCGrid?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor

 

[Assimilator1]

Thanks for the info Skillz, I was wondering where else I could find the info!

 

[crashtech]

Thanks for the info Skillz, I was wondering where else I could find the info!

They have a Facebook page and a Twitter presence, if you can stomach that kind of thing:

Bei Facebook anmelden

Melde dich bei Facebook an, um dich mit deinen Freunden, deiner Familie und Personen, die du kennst, zu verbinden und Inhalte zu teilen.

www.facebook.com

https://twitter.com/WCGrid

 

[waffleironhead]

Thats why my potato fleet was inactive. Had to switch them over to asteroids for time being. Hopefully they get it sorted.

 

[crashtech]

The most recent news on WCG's storage array problems, most recently edited (update #5, presumably) 2 days ago:

Hello everyone, hope you had a great weekend. We are still working with data centre to resolve the hardware failure so we can restart the storage, BOINC and website ASAP. We will post updates as we receive them. Thank you for your patience.
Update: Unfortunately, additional hardware problem on the storage server besides the RAID card are preventing us from restarting. Working with the data center on the alternative solutions.
Update #2: Unfortunately, the RAID controller was not the root cause of our storage system failure, the PCI bus failed. Data center is in the process of moving the disks to an alternate system and we will post updates as we progress. Once again, thank you for your patience.
Update #3: As of this morning, the data center continues to work on booting the temporary replacement DSS 7000 storage system. They are attempting multiple alternative strategies to resolve current failures.
Update #4: The "new" system did recognize the data hardware RAIDs. All have been rebuilt, and the data center is attempting to repair the OS drives/RAID.
Update #5: The storage server was revived yesterday late afternoon. Both database filesystems mounted as before, but the science filesystem did not. It needs a repair; erasing the old log first.

 

[Markfw]

So, yesterday, I checked and it looked like it was back up. So I added a project to BOINC using WCG, so it looked to be up. But its still waiting for a unit ??? Still a problem.

 

[Assimilator1]

Yea I've still got a bunch of tasks to upload.

 

[crashtech]

They've put some info up on the WCG site:

Hardware Recovery Update

The website has been restarted and we are working on rebuilding the science database so BOINC can restart soon.

www.worldcommunitygrid.org

 

[Markfw]

They've put some info up on the WCG site:

Hardware Recovery Update

The website has been restarted and we are working on rebuilding the science database so BOINC can restart soon.

www.worldcommunitygrid.org

Thanks. When it first went down, there was no way to get to that forum, so I was using facebook updates. Those stopped a few days ago. This is really a pisser.

 

[crashtech]

Ran across this today, I may have missed some previous updates:

New hardware
While we prepare the new and improved hardware to host our databases and parallel filesystems, we have been using a temporary system provided to us by the data center. All data is confirmed intact and there has been no data loss as we continue to recover. The recovery system is a stand-in for the storage server that failed, selected for hardware compatibility to recover the data. We will not be continuing with the recovery system indefinitely, and it will be discontinued only once the new storage system has been fully installed and synced with the recovery system for a smooth handoff.

BOINC database is UP
The BOINC database is now up and running, joining the website/forums database which has been up since last week. However, upload/download of workunits is paused until we restore the parallel filesystem that supports the workunit management stack, to the state it was in at the time of the hardware failure. Deadlines have been extended and valid results computed during this pause will be credited when we resume.

Website crashes
During the hardware recovery process the website has been intermittently crashing. Looking into the cause we identified bugs that only present themselves in such cases as the BOINC database being offline, and other resources unavailable as we recover the system. The website will now remain available to users in these cases or restart automatically after crashing.

In the meantime, we have posted research updates from the ARP and MCM teams. We are planning on sharing more updates soon.

If you have any comments or questions, please leave them in this thread for us to answer. Thank you for your support, patience and understanding.

WCG team

World Community Grid - View Thread - Hardware Recovery Updates (Updated March 31, 2023)

 

[Markfw]

Ran across this today, I may have missed some previous updates:

World Community Grid - View Thread - Hardware Recovery Updates (Updated March 31, 2023)

I still can't get any units.....

 

[crashtech]

I still can't get any units.....

Yeah, and my uploads are still stuck. It doesn't appear that this project has the budget or sense of urgency at Krembil that would be necessary to return it to its former glory...

 

[Markfw]

Yeah, and my uploads are still stuck. It doesn't appear that this project has the budget or sense of urgency at Krembil that would be necessary to return it to its former glory...

Due to 1) my electric bill and 2) the lack of cancer research places that I will support (nothing to do with Russia) I am down to 8 computers running. But at least I have 60-70 mill ppd going for F@H. 4 5950x and 4 7950x configured for the power of 142 watts the same as 5950x. One 5950x will soon be replaced by a 7950x@142 watts, do it will be 3 and 5, with 2 idle 5950x's idle and for sale !

 

[cellarnoise]

Due to 1) my electric bill and 2) the lack of cancer research places that I will support (nothing to do with Russia) I am down to 8 computers running. But at least I have 60-70 mill ppd going for F@H. 4 5950x and 4 7950x configured for the power of 142 watts the same as 5950x. One 5950x will soon be replaced by a 7950x@142 watts, do it will be 3 and 5, with 2 idle 5950x's idle and for sale !

Well maybe more medical than cancer (and with little Russia ties that I know of), but TN-Grid and Denis have tasks occasionally. They have been more on than off recently....

Not so much for you Mark, but for others that have not ran these projects before:
Denis

Project status

Official web page of the volunteer computing project DENIS@Home. This project is developed using BOINC and maintained by the CoMBA research group at the University San Jorge (Zaragoza, Spain)

denis.usj.es

TN-Grid:

Project status

And Mark continues to run Rosetta of course :

Project status

Carry ON!

 

[Markfw]

Well maybe more medical than cancer (and with little Russia ties that I know of), but TN-Grid and Denis have tasks occasionally. They have been more on than off recently....

Not so much for you Mark, but for others that have not ran these projects before:
Denis

Project status

Official web page of the volunteer computing project DENIS@Home. This project is developed using BOINC and maintained by the CoMBA research group at the University San Jorge (Zaragoza, Spain)

denis.usj.es

TN-Grid:

Project status

And Mark continues to run Rosetta of course :

Project status

Carry ON!

Yup, at the moment, its Rosetta and F@H.

I do miss the competition, but I don't miss the "lets massacre your teammates using the cloud(in an inter-team competition)".

 

[Skillz]

I do miss the competition, but I don't miss the "lets massacre your teammates using the cloud(in an inter-team competition)".

Except that didn't happen.

 

[Markfw]

Except that didn't happen.

It absolutely HAS happened before, just not this last race in Jan, and thats probably due to my complaining.

 

[Skillz]

It didn't happen last year either.

 

[Markfw]

It didn't happen last year either.

Mr know-it-all, if you want to keep arguing, maybe I should quit the team altogether, since you seem to speak for it now.

Edit: and YOU are the reason that I quit competition.

 

[Assimilator1]

I don't believe Skillz has ever claimed to speak for the team, and their are plenty of us with our own voices, so I'm not sure where you get that idea from.

I don't know (or recall) what the situation was regarding your statement that icloud was used in competition, IIRC it isn't (or wasn't at least) a banned practice in races.
That said, I'm on the fence as to whether it's fair. Is it any different than buying new computer gear? I suspect many of us who haven't commented about it are because of these uncertainties.
I vaguely recall that some of us discussed this in another thread, perhaps someone could link me to it if all angles were covered there? (was it the F@H race thread?[edit] Yep, from here, at a glance [edit 2] Nope, that wasn't the whole conversation, but more here). Failing that, I'd rather we didn't carry it on in this thread, but perhaps start a new thread if wanted?

Btw, I think it's rather off of you that you threaten to leave the team because of a disagreement with a single (?) team member.

*****************************************

Going back to the WCG problems, I've still got a bunch of tasks to upload, I assume they're not fully back up yet? Or is it just me? lol

 

[crashtech]

@Assimilator1 and all, here's the latest:

March 27, 2023 update

Data transfer to new storage system
We have started transferring all data from the recovery storage unit to our new storage system on Friday. Based on the current rate of transfer, we expect to have all data transferred/verified later this week. We will then download all processed WUs, after which we can resume sending work units to volunteers. We plan to start with MCM and OPN/OPNG; followed by ARP and then the new SCC work units.

In the meantime, we have confirmed that our daily database backups for BOINC and for the website/forums are working. The databases have been recovered and transferred to the new, faster storage already. Incremental backup to tape archive has been implemented on the new storage.

 

[biodoc]

Well, that seems promising.

 

[crashtech]

Yeah, I haven't looked for a bit but I think my uploads are still stuck. :|

 

[Assimilator1]

Yep, mine are still stuck too.
From their forum :-

March 31, 2023 update

Data transfer update & maintenance check
Earlier in the week, we ran into HDD failures while transferring the data from recovery storage to the new storage system. The issue has since been resolved and we have resumed transferring data, expecting to finish it by 5pm today. At 4pm, we will be conducting a brief maintenance on the website and forums to transfer the DB2 filesystems to our new storage system, which will result in restricted access to the website for up to 30 minutes. If all goes well, this could be the final step towards the full storage system upgrade.

We evaluated the possibility of starting download of processed WUs, while not sending new WUs out. It was determined that the risk of complications that might result from doing this with incomplete information available to our scheduler and BOINC or any other unforeseen issues is too high. We have extended the deadlines for workunits that were processed and await upload to WCG.

While we wait for the data transfer to finish, we are working on resolving other long standing issues such as device recognition.