RaptorCS POWER9 Blackbird PC: An expensive mistake

Published 2019-09-23 on Drew DeVault's blog

November 2018: Ordered Basic Blackbird Bundle w/32 GB RAM: $1,935.64

Update 2019-10-10: RaptorCS’s redemption: the POWER9 machine works

June 2019

Order ships, and arrives without RAM. It had been long enough that I didn’t realize the order had only been partially fulfilled, so I order some RAM from the list of recommended chips ($338.40), along with the other necessities that I didn’t purchase from Raptor: a case ($97.99) and a PSU ($68.49), and grab some hard drives I have lying around. Total cost: about $2,440. Worth it to get POWER9 builds working on builds.sr.ht!

I carefully put everything together, consulting the manual at each step, plug in a display, and turn it on. Lights come on, things start whizzing, and the screen comes to life - and promptly starts boot looping.

June 27th

Support ticket created. What’s going on with my board?

June 28th

Support gets back to me the next day with a suggestion which is unrelated to the problem, but no matter - I spoke with volunteers in the IRC channel a few hours earlier and we found out that - whoops! - I hadn’t connected the CPU power to the motherboard. This is the end of the PEBKAC errors, but not the end of the problems. The machine gets further ahead in the boot - almost to “petitboot”, and then the display dies and the machine reveals no further secrets.

I sent an update to the support team.

July 1st

We have normally only seen this type of failure when there is a RAM-related fault, or if the PSU is underpowered enough that bringing the CPUs online at full power causes a power fault and immediate safety power off.

Can you watch the internal lights while the system is booting, and see if the power LED cluster immediately changes from green to orange as the system stops responding over SSH?

The IRC channel suspects this is not related to the problem. Regardless, I reply a few hours later with two videos showing the boot up process from power-out to display death, with the internal LEDs and the display output clearly visible.

July 4th

“Any progress on this issue?”, I ask.

July 15th

“Hi guys, I’m still experiencing this problem. If you’re unsure of the issue I would like to send the board back to you for diagnosis or a refund.”

July 25th

Sorry for the delay. Having senior support check out the videos.

Thanks for writing back. We should have something for you by tomorrow during the day.

July 31st

Hi Drew.

The videos are being reviewed this week. Thank you for sending them.

Please stay tuned.

September 15th

No reply from support. I have since bought a little more hardware for self-diagnosis, namely the necessary pieces to connect to the two (or is it 3?) serial ports. I manage to get a log, which points to several failures, but none of them seem to be related to the problem at hand (they do indicate some network failures, which would explain why I can’t log into the BMC over SSH for further diagnosis). And the getty is looping, so I can’t log in on the serial console to explore any further.


That was a week ago. Radio silence since.

So, 10 months after I placed an order for a POWER9 machine, 3 months after I received it (without the RAM I purchased, no less), and over $2,500 invested… it’s clear that buying the Blackbird was an expensive mistake. Maybe someday I’ll get it working. If I do, I doubt the “support” team will have been involved. Currently my best bet seems to be waiting for some apparent staff member (the only apparent staff member) who idles in the IRC channel on Freenode and allegedly comes online from time to time.

I’m not alone in these problems. Here are some (anonymized) quotes I’ve heard from others while trying to troubleshoot this on IRC.

On support:

ugh, ddevault, yeah. [Blackbird ownership] has not been a smooth experience for me, either.

my personal theory is that they have really bad ticket software that ‘loses’ tickets somehow

On reliability:

I’ve found openbmc’s networking to be… a bit unreliable… maybe 20% of the time it does not responed[sic]/does not respond fast enough to networking requests.

yeah the vga handoff failing doesn’t surprise me (other people here have reported it). but the BMC not getting a DHCP lease is odd. (well maybe not that odd if you look at the crumminess of the OpenBMC software stack…)

So, yeah, don’t buy from Raptor Computer Systems. It’s too large and unwieldly to be an effective paper weight, either!


Erratta

2019-09-24 @ 00:19 UTC: Raptor has reached out and apologized for my support experience. We are discussing these problems in more detail now. They have also issued a refund for the unshipped RAM.

2019-09-24 @ 00:51 UTC: Raptor believes the CPU to be faulty and is shipping a replacement. They attribute the delay to having to reach out to IBM about the problem, but don’t have a satisfactory answer to why the support process failed. I understand it’s being discussed internally.

2019-09-24 @ 13:08 UTC:

After investigation, we are implementing new mechanisms to avoid support issues like the one you experienced. We now have a self-serve RMA generation system which would have significantly reduced your wait time, and are taking measures to ensure that tickets are no longer able to be ignored by front line support staff. We believe we have addressed the known failure modes at this time, and management will be keeping a close eye on the operation of the support system to ensure that new failure modes are handled rapidly.

They’ve tweeted this about their new self-service RMA system as well:

We’ve made it easy to submit RMA requests for defective products on our Web site. Simply go to your account, select the “Submit RMA Request” link, and fill out the form. Your product will be warranty checked and, if valid, you will receive an RMA number and shipping address!

— @RaptorCompSys via Twitter

I agree that this shows positive improvements and a willingness to continue making improvements in their support experience. Thanks to Raptor for taking these concerns seriously. I hope to have a working Blackbird system soon, and will publish a follow-up review when the time comes.

2019-10-08 @ 22:30 UTC A source quoted anonymously in this article asked me to remove their quote, after a change of heart. They feel that the attention this article has received has made their statement reach beyond the level of dissatisfaction they had with Raptor at the time.

Have a comment on one of my posts? Start a discussion in my public inbox by sending an email to ~sircmpwn/public-inbox@lists.sr.ht [mailing list etiquette]


Articles from blogs I follow around the net

Go Turns 10

Happy birthday, Go!

via The Go Programming Language Blog November 8, 2019

Working full-time on open-source software

I’ll soon be working full-time on open-source software! I’m pleased to announce that I’m joining Sourcehut. Huge thanks to Drew DeVault for making this possible. I also want to thank everyone supporting Sourcehut and allowing it to grow. Being able to do …

via emersion October 15, 2019

Updates in September 2019

This post gives an overview of the recent updates to the Writing an OS in Rust blog and the used libraries and tools. I finished my master thesis and got my degree this month, so I only had limited time for my open source work. I still managed to perform a…

via Writing an OS in Rust October 6, 2019

Generated by openring