au QuaStation Part 11 - The QuaStation Ubuntu Project

 As I mentioned in my last post about the QuaStation Linux Kernel, there is a QuaStation Ubuntu Project, which means something more customized than the vanilla Ubuntu with Banana Pi kernel that I have been running.  

The question is, then, what is different about this userland compared with the stock Ubuntu userland I have been running?

Well, let me summarize the main points from author's readme.md in English, with my comments in italics.

First, a summary of the system itself.

About the CPU/SoC - The Realtek RTD12955 (4 core)

  • The CPU itself has H.264 encoding and decoding, but unfortunately the required OpenMAX library is proprietary, so it doesn't work in Linux
    • He goes on to talk about some leaked ffmpeg with OpenMAX support.
    • The Ubuntu version he created includes some OpenMAX library from openWRT?
    • Not sure how useful this would be - I suppose if you wanted to do something like rip and encode DVDs from the QuaStation.   
  • There is a GPU on the SoC, but since the QuaStation has no video output, that is a bit pointless.  
    • I mentioned before my last post that someone used the screengrap utility in the stock Android OS to take a screen shot long ago.  I agree, not much point, though, unless you can use this somehow for screen sharing the "physical" console.  
  • Compared with Allwinner, Amlogic, and Rockchip, there are fewer devices employing Realtek SoCs, so there is less information available.
About the CPU Fan

  • The Banana Pi kernel doesn't recognize the fan, but the author managed to dig through the stock device tree in the eMMC  to fix this, and says it runs much quieter now.  
  • On the one hand, I have like 20 of these machines, mostly running in the same spot, and the noise doesn't bother me - but on the other hand, proper fan support is fantastic news.
RAM: 2GB (Samsung ram)

  • The linux kernel and Memory mapped IP take up about 400MB, so there is only about 1.6GB remaining.
  • Not sure how this relates to the 1.6 I mentioned before, as that number showed even in uboot.  On normal PCs, the actual memory includes the amount taken by the kernel.  f.e. we don't say an 8GB Intel laptop has 7.6GB, even though the kernel obviously takes up some of that memory.  Does the memory mapped IO need to take that much?  I wonder how much is for the graphics frame buffer.  This may be something for me to test.  
eMMC ("ROM") - 8GB (Samsung)
  • This contains the stock Android kernel.
  • He provides a link to the contents here.
  • This might be useful if for some reason you wanted to revert to stock after overwriting the original eMMC.  Note that you don't have to overwrite the eMMC in order to run your own OS unless you want to.  You can always back it up yourself and/or copy from another machine.
HDD - 1TB HGST

  • This is a 2.5 inch HGST disk, which can be removed relatively easily.  
WiFi
  • Two Realtek Chips
  • RTL8812AR
    • Connected to the internal PCIe, shows up in lspci.
    • Rates as supporting 5Ghz and 2.4Ghz both, but only shows 2.4Ghz aaccess points during a scan.  This may be a driver issue.
    • Seems a bit slower and suseptible to interferance than a normal PC wifi card.
    • This is used by the stock Android system to create the 5Ghz access point.
  • RTL8192ER
    • Also connected via internal PCIe, also shows in lspci.
    • Only supports 2.4Ghz, but supports WiFi 10n, so it is pretty fast.
    • This is used by the stock Android system for the 2.4Ghz access point.  
  • Again, for me the WiFi coming on when I didn't want it to, and different devices taking the same MAC address was a problem, so most of my time dealing with WiFi has been with how to reliably keep it reliably turned off without simply removing the driver and recompiling the kernel.  
Bluetooth IC: Realtek

  • This is connected via Serial internally, and shows up as /dev/ttyS1
  • You need to run a separate utility to make this appear as a bluetooth device, which he has included in his Ubuntu script.  
  • It will only run with the firmware from the stock Android in the eMMC.
  • He has also modified the device tree to make sure it works even after rebooting.  
  • This could be a really useful tool if you can set up an easy way to get a console over bluetooth - then even if ethernet IP networking is down for some reason, you could connect and fix it.  Otherwise, not sure how much use it would be for a server type device.  
LTE Modem
  • This shows up as a TTY over USB if USB-ACM is enabled in the kernel.
  • You can check that it is recognized via lsusb.
  • This is a Qualcomm LTE model, but the model number indicates that it may be a special model just for the QuaStation.  It only handles the frequency bands handled bu au telecom.  (KDDI)
  • He checked that the "AT" modem command works, but he didn't have a SIM card, so hasn't verified if transmission actually works.  
  • Supposedly it works with Rakuten Mobile MNVO SIM Cards (as per the guy selling units with Plex pre-installed), and the chip has its own Pocked WiFi style web interface (Source)
  • Interesting, but not sure how useful.  Again, it might be a good say to get into the box if you get locked out, or just to be able to log in if you are outside the home - but if you have a number of these, then even the cheapest LTE plan will add up.  For example, I saw low speed LTE SIMs for like $3 a month, but $3 x 10 servers is $30, for example.  Still, it's good to know that it works.  
USB Ports

  • One USB 3.0 port, One USB 2.0 Port
  • He mentions issues with the USB 3 port only showing up as USB 2, but said it was fixed by modifying the device tree.  I never had this issue (it shows up as 3.0 on all the kernels I run).
  • This means the author's "normal" QuaStation build might not work with USB3?  Something to check.  This is an important thing to me, because this is the only high speed port on these devices other than the SATA.  You need USB3 for fast wired networking or external drives.  
SD Card Slot

  • This shows up as /dev/mmcblkX (where X is a number
  • The eMMC also shows up as mmcblkX, so be careful not to confuse them.
  • This is unlikely to happen, as the embedded MMC has a lot of partitions, and most SD Cards don't, but then again if you clobber the MMC and overwrite uboot, you won't be able to recover the device.
GPIO Buttons (Power, Reset, WPS, Copy)

  • These are the buttons on the outside of the unit.  
  • He was able to get them working by digging around the device tree.
  • The script needed to query this button is listed as GPL, but no souce provided, but he managed to reverse engineer a script.  
  • He is able to trigger events by pushing the buttons.
  • He added a script so if you press the power button for over 3 seconds, it will shut down the system.  (Techincally, set the ubuut power flag to "off" and then reboot the system).  
  • This is fantastic, because if you think the device is frozen you can now try to power it down properly.  Until now, I just had to pull the power plug.  In some cases, it wasn't dead, just disconnected from the network!
  • Reset button - This is set to just perform a normal reboot. (i.e. shutdown -r now?)
  • WPS (WiFi) button & Copy Button - He didn't have any use for these, so he set them to blink the LEDs.  This is actually pretty nice, since it is an easy way to tell is the machine is alive.  You could also set these buttons to copy data from an external USB drive (like the original functionality of the copy button), rip a CD or DVD, reset networking settings, or anything else interesting like a self destruct button to  wipe the drive!  Until now, the buttons have been useless, so this is a cool feature for me.
LED Lights (Power, LTE, WLAN, HDD, Copy)

  • He was able to get them working by digging around the device tree.
  • If you enable it in the kernal, the default driver is enough to make them work.
  • He has a command in the /etc/rc.local to set the color of the power LED on boot.
  • For the LTE, WLAN, and HDD LEDs, he has set them to light up if you press the WPS button.
  • I would think these would be easy enough to have light up when the relevent device is accessed.  Particularly the HDD one would be useful.
  • For the Copy LED, he set it to light up if you press the copy button, and turn off if you hold the copy button for 3 seconds.   

  •  Linux Kernel Selection

    At the moment, there are basically two choices:

    • 4.9.119 (QuaStation-Kernel-BPi) - This is a kernel he improved slightly from the Banana Pi W2 kernel.  (The same one I am using for my newer machines)
    • 4.1.17 (QuaStation-Kernel) 
      • This is the one from KDDI released under GPL, but he modified it a little to work better in Ubuntu.
      • This is almost the same as the one in the stock Android OS, but the device tree results are different, and the second PCIe slot is not recognized, etc.,  so the author thinks that this source is likely a bit old, and was the source for a development board or slightly older unreleased model.  
      • The U-Boot source code has also been released (here).  This was released 4 years ago, but only recently made it onto GitHub.  
    Both versions are heavily customized for the Realtek SoC, including support for Realtek device drivers and the device tree, etc.  The mainline kernel doesn't have the Realtek drivers, so it won't work. 

    4.9.119 is considered to be an old version of the kernel now, but it would be a lot of effort to get a newer version of the kernel working (and as a general rule, SoC vendors don't update their ports).  Basically, there is no alternative to using something a bit old here.  

    He mentions that Ubuntu 20.04 LTS works fine, as does Docker.

    Since I am Using Ubuntu 20.04 LTS on the machines with the 4.9 kernel, I already knew this...)

    Basically, he recommends using 4.9.119 because, the 4.2.17 based kernel is not only even older, but the second PCIe slot  where the 5GHz WiFi chip is connected to is not recognized due to an error.  Also, he has not battle tested 4.1.17 as much, so it may not be as stable, etc.  

    Interesting, since the Plex guy recommends the older kernel.  He also turns off the WiFi on the units he ships, due to worrying too much about bogus claims about WiFi liability.  

    So What?, you ask

    So the summary above is all well and good, but you might ask yourself "Okay, so basically, we can use the buttons now, and control the LEDs.  The LTE moden and CPU fan control are working too.  None of that seems terribly useful"

    CPU Fan Speed Control

    The CPU fan, honestly, is not a big deal.  Even on the max speed it isn't very loud, to the point that I never realized that the speed control wasn't working.  The amount of power it uses must be very trivial as well.  Probably the biggest benefit to proper control is that it may take longer to wear out if being run at a slower speed.  

    LTE Modem and mSATA Slot

    As for the LTE modem, there might be specific use cases where it is actually useful.  The original idea was that this device would be purchased and set up in the owner's house.  The device would connect via LTE and set up a WiFi access point.  It was assumed (fairly enough in Japan) that the owner may not have home internet, so using WiFi means you could back up photos from your phone at a relitely high speed when you were at home.  Having LTE means that you could access the photo library even while you were away from home.  

    As such, if you don't have internet at home, keeping the LTE could be a handy way of accessing this device from outside.  If you do have internet, though, you would be much better off using that instead of LTE if you plan on accessing it on the go very often.  Even if you have a plan that allows 100GB per month, it would take 10 months to fill up a 1 TB drive.  (And you would have to pay quite a bit of money for the privlidge!).

    If we are being honest, this device was really a way to sell LTE subscriptions.

    There are some very interesting LTE plans, such as ones with a static global IP and a cheap monthly fee that would let you have reliable access into the server from anywhere in the world.  Example: IPSIMServersMan SIM LTE.  At the moment, the IPSIM seems to be the better deal, as it will give you an "unlimited" data plan with a public, static IP for less than $10 per month.  This is limited to 200kbps on some plans, but that's fine for something like SSH.  In fact, it's even fine for something like apt-get upgrade, it will just take some time.  The major problem is that both of these services are just MVNOs for NTT Docomo, much like OCN Mobile One.  

    The model of LTE modem in the QuaStation only supports au (KDDI) bands.  In order to get a cheap au compatible SIM, you would need to use Nuro MobileMineo or IIJmio, or one of the other au MVNOs like Aeon mobile, etc.  There are still plans for less than $10, though I am not sure they have public or static IPs.  You could use a VPN service like Tailmode, etc., but you could do that with WiFi anyway.  Another potential issue is that you might need to pay this fee for each device (if they were in different location), which could quickly add up.  Point being, whether or not this is worth using depends heavily on your use case.  If you were using these devices as distributed data backup to locations without internet, then maybe.  

    The biggest revelation for me is that fact that the LTE card is in fact removable - which means that you can replace it with a different mSATA card!  

    Sure you could swap out the au LTE modem for one that can handle Docomo or Softbank, but you are just adding more expense for something only needed in a minority of use cases.  

    In reality, the main candidate would be an mSATA SSD drive.  This would allow for a root filesystem drive that is faster and more robust than USB or an SD-Card, without using up a USB port, so the following becomes practical:

    1. Boot - USB on USB2 port

    2. Root - mSATA

    3. User/Home - SATA

    This way, you can keep the OS off of the HD and make it easier to swap, without putting it on flash memory that lacks wear leveling.

    mSATA drives are actually more expensive than m.2 drives, but the lowest capacity drives (like 30GB) are still quite cheap.  Sure, a $15 drive might double the cost of your QuaStation if you got it super cheap, and sure, a USB drive can be found for a bit cheaper, but mSATA is a great way to add a reliable root drive to the QuaStation that will make it easier to swap out the HDD when needed.    USB drives tend not to have wear leveling, and therefore aren't actually great for root filesystems that will have things like constant log file writing going on.  

    I have ordered a 64gb Toshima mSATA SSD, and so I will be testing this out soon in order to see if it actually works in the QuaStation's slot.  

    As for what else you could use the LTE cards for, I am not sure.  One interesting fact is that the SIM card socket is on the device itself, not the mSATA card.  This means that even if I stuck it into another random computer with an mSATA slot, I would have no way to insert a SIM card.  

    LED and Button Control

    Here is where we come to the truely useful (and free!) part.

    As a headless device, it's difficult to determine what is going on when you can't access a QuaStation.  You try to SSH in, and it doesn't connect...

    • Is SSHd running?
    • Is the machine connected to the network?
    • Does it have an IP address?
    • Did the kernel panic?
    • Is it actually booted up at all?
    • Can it find the network card?
    You can try a few things, but eventually you just have to rip out the power, plug it back in, and hope it comes back up.  Even if it does, you don't have an easy job to figure out why it died.  

    Worse yet, most of the time, it probably didn't crash.  Linux is remarkable stable.  That means you forced the power off on a running system.  If you were using ext4, that's very bad, and may cause your machine not to boot.  If you were using btrfs, it's not quite as bad, but it's still not great.  

    As an example, I have found that avahi sometimes just stops working.  I try something like "ssh quastation17.local" and get something like "hostname not found" - but then I can see see it's up and running from the ResilioSync peers dialog on another computer.  If I knew the IP then I could connect to it with something like "ssh 192.168.1.117" - no issues!  You can set up statis DHCP assignments, or just not use DHCP, but 

    So the first thing is, it would be great to have some kind of heartbeet signal so you can see if the machine is running or crashed.  

    I set up a script that blinks the 3 LEDS on the back in sequence once per minute.  This by itself is amazing, because now for any machine that doesn't respond, I can just stare at it for a minute or two to see if the lights blink.  If they do, I know it's still up and running.  If not, I will know not to feel guilty about pulling the power cord.

    More to the point, you don't have to pull the power cord, becuase the power button now actually does something!  You can initiate a controlled shutdown even if you don't have network access.  Just from whether it responds to this or not, you can also know whether or not the device if actually running.  

    besides the heartbeat script I made, it should be possible to add all sorts of troubleshooting indicators. There are 5 LEDs to play with, and each one supports 3 colors: Red, Green, and orange (from lighting up both red and green).

    For example, you could have a ping test that tried to ping your router once every 10 minutes, and if it fails, then the "copy" LED turns red.  If it passes, the LED turns green.  Maybe if it can reach your router but not do a DNS resolve for ubuntu.com, then it turns orange.  So now the copy LED is the "Network status" LED.

    The power LED can be set to green after boot up, unless there is a major issue.  If you are more than 95% full on any disk, it can be set to orange.  If it can't mount the IDE drive or find the network connector, it can be set to red.  

    ... and we still have 3 more LEDs.


    Then there are the GPIO buttons.  Let's look at that they originally did.  

    On the back:

    • Power - This is "fake" in that it only tells the uboot firmware to run the OS, or tells the OS to exit back to uboot.  (i.e. the board never actually turns off)  This proably makes sense to actually use as power.  
    • Reset - This also may make sense to use for its intended purpose, and works will with this new patched kernel/ubuntu.
    • WPS - The author set this to blink the 3 LEDs on the back, which is actually good because it also gives an indication as to whether the device is working, without even ahving to wait a minute for the heartbeat - but it could be set to do something even more useful.  This could also of course bet set to do what was originally intended -but since I don't think using WiFi is ideal for a device with 1TB or storage, it would probably be better to repurpose it for better things.  
    • Copy - This was intended to initiate a copy of data from the phone to the devices internal HDD, or from the internal HDD to the external backup HDD (I forget which).  This could, theoretically still be useful for copying to a backup drive, but I suspect cron would be the better choice for that.  
    What other uses could we have for the WPS and Copy buttons?

    Pressing the WPS button could run a network reset script that restarts avahi and does an ifdown/ifup, followed by dhclient (if appropriate).  Doing a long press could restart SSHd and other relevant services.  This way, if the device is not responding, you can press this button to try to fix it and get in.  

    If you wanted to use the QuaStation to store photos from your digital camera, you could set the copy button to copy all files on any inserved SD Card or USB stick to a certain folder on the internal HDD.  

    If you wanted to use the QuaStation as a CD/DVD/BR ripper, you could plug in an external drive, and set the copy button to run cdparanoia or whatever.  it could rip the files and then put them in a network share.  

    You could set up the copy button as a self destruct option.  If you hold down the button for more than 10 seconds, it will wipe the drive!  Great for when the G-men are banging on the door.  

    You could set up the copy button to mount any inserted USB stick, copy a certain file from it, (or some server) and then run that file as a script.  This could allow you to give commands to the device even when you can't get in through the network.  

    For me the olnly issue with the copy button is that it's on top, so not accessible if you have stacked a lot of the QuaStations as I have.  

    For the reset button, maybe a long press resets the root password and changes the network setting to DHCP.  

    If you really wanted to get fancy, you could set up a script where if you press the buttong in a certain combination or order it runs something.  

    The main point is that for a headless device like the QuaStation, the combination of button control and LED indicators means that you have some quick and easy control and a window into what is going on even if you can't get into the box by other means.  





    Comments

    Popular Posts