over the mount’n

Okay, that was a terrible pun. I thought I’d share something today which is pretty cool although it gets a little technical at the end. You can actually edit the files on a tiny microSD card like you might have in some device like a 3D printer or similar. And you don’t have to have it running (like in your product/device) in order to do so.

Smart Things (IoT)

We live in a world of ever-smarter things. There’s a new term, the Internet of Things (IoT) which basically means that lots of the products we’re using now not only have little computers inside, but they also use the Internet in some way.

Tiny Hard Drives

Given the diminutive size of some of these new products, they often use the microSD card as an actual hard drive. These days, they include an entire operating system which boots up the product’s computer and provides a variety of functionality.

microsd

The Bleeding Edge

For some of us, we like to install the very latest version of software for something like this. Those updates sometimes come in the form of an entire image file for the microSD card in this case.

Instructions for editing files on an image file itself

Okay, so I have downloaded an image for a Raspberry Pi computer and the version name is Jessie. The name of the operating system itself which runs on this computer is called Raspbian, btw. The image’s filename is then raspbian-jessie.img, easy enough.

I’m running these commands on a Linux-compatible computer:


$ file raspbian-jessie.img

raspbian-jessie.img: DOS/MBR boot sector; partition 1 : ID=0xc, start-CHS (0x0,130,3), end-CHS (0x8,138,2), startsector 8192, 129024 sectors; partition 2 : ID=0x3, start-CHS (0x8,138,3), end-CHS (0x213,120,37), startsector 137216, 8400896 sectors

So we’re interested in only one number here, that value after startsector for partition 2, namely 137216. Multiply that by 512 to get 70254592 which we use in the next command:

$ sudo mount raspbian-jessie.img -o offset=70254592 /mnt

This means, essentially, “open up the file indicated (at an offset of 7 million or so characters into that file) and show me everything that’s in it and put that in the /mnt folder area”.

So now, you can actually edit the image’s computer name in the /mnt/etc/hostname file, for example. Assuming we have done so, we now unmount the file:

$ sudo umount /mnt

And you’ve managed to edit its internal files in place! The typical activity next is to burn that image to another microSD, put it into your smart refrigerator’s computer or whatever and boot it up.

Conclusion

Okay, so you’re not as excited as I am but this is a giant leap forward in workflow for me. Since I maintain several of these image files, this is pretty cool stuff.

Imagine if you were trying to build a supercomputer with a hundred individual (small) computers and it were necessary to then build 100 different image files each with their own setup. This would be so not fun if you had to do this manually. Using this new method, you could script all this and then run it.  By the end of some lengthy process, you would have all hundred different image files as produced by this method. Huzzah…, right?

Advertisements

too much fun

My two packages arrived today at the post office so I just hauled in all the loot from this earlier post in which I’ve purchased some new toys.

Raspberry Pi Zero W

The photos from their website don’t really describe how truly small this computer is now. They’ve somehow managed to stack the RAM on top of the microprocessor to save space. As I’ve apparently ordered the wrong video adapter cable, I’ve got a trip over to Best Buy Frye’s Electronics this evening so that I can sort that one out. I need a female HDMI to DVI, in other words. Otherwise, I’m still pretty stoked. Since there’s only one micro-USB I think I’ll temporarily need a small USB hub while I’m at it.

PiZero

NeoPixel Ring

This arrived as well, all four of the segments but it was lost on me that I’ll need to solder each of them together. Fortunately, I have a soldering iron here somewhere. :looks around: I’m certain of it.

COZIR CO2 Sensor with RH/Temp

And in the other relatively BIG package is the relatively small sensor package. No wonder they charged me $21.88 to ship this to me. Seriously, it weighs about an ounce.

And it looks like I’ll need a 2×5 jumper to attach this over to the Raspi, with a solder-able header for that, too.

Update 1

Alright, I’m back from Frye’s with a handful of stuff and I’m back in business. The video adapter allows me to see what’s coming out of the Raspberry Pi Zero W and the micro-USB hub allows me to hook up a keyboard and mouse to talk to it locally. A first install with the Raspbian Jessie Lite image resulted in a terminal-only configuration (I must have been in a hurry and didn’t read the differences on their page) so a second install of Raspbian Jessie with Pixel was just what it wanted: a full desktop experience.  If I get some time this weekend I’ll try to have it talk to either the sensor or the light ring.

Update 2

I just managed to solder together the NeoPixel ring. Due to the size of the electrical pads on the ends of these, I’d suggest that this falls into the catagory of advanced soldering and not to be taken on by the average person.

NeoPixel
These are not my lovely hands.

Additionally, I’d say that this feels a bit fragile in the area of the soldering joints between each quarter-circle. I’m going to suggest that anyone who incorporates one of these into their project needs to seriously think about ways of making this more stable/reliable since the soldering joints between them are tenuously-small.  (Imagine three distinct electrical connections across the tiny width of this thing.)

What I also found is that there isn’t anywhere to clamp a hemostat for soldering these jumpers since the LEDs run all the way to the end where the connections should go.

I did add an inline resistor as Adafruit suggested to lower the input voltage or perhaps to lower start-up voltage spikes.

I managed to re-purpose a nice external 5V switching power supply that should drive all the LEDs nicely. It was left over from the supercomputer project when I swapped in a USB-based charger instead for that. Amazingly, Adafruit suggests that those 60 LEDs need a whopping 3.6A of power to drive them. I’m guessing that reality is more like 1A but I’ll play this safe. Per Adafruit’s suggestion I included a 1000 µF electrolytic capacitor across the output voltage to protect the NeoPixels.

VGD-60

So I’m prepped to do a final test of the NeoPixel ring for power and functionality on a standard Raspberry Pi 3 rig (since it sports an actual header). Once I’ve coded a test and verified that it works then I’ll take the soldering iron to the Raspberry Pi Zero W and wire it in with a quick-connect.

headerwire

I’ve now got the Raspberry Pi Zero W booting with just the power adapter. Note that you can rename its hostname, toggle on the VNC Server, adjust the default screen resolution to your liking and then—in the Finder program in OS X—open up a remote session to its Desktop with vnc://pi@hostname.local, for example. Or, toggle on the SSH Server and connect from a Terminal session with ssh pi@hostname.local.

Have I mentioned how awesome it is to have a fully-functioning computer for $10 (plus $6 for the micro SD)?

And now the power supply is completed and wired to the NeoPixel ring. Everything’s set for 5V DC in at the moment but I may try to adjust the input voltage down to 3.3V later for technical reasons. (The NeoPixels are designed for the Arduino and its output data voltage is 5V whereas the Raspberry Pi is only 3.3V. By adjusting the input voltage down then it makes a 3.3V data line look bigger than it is. There are other tricks like adding a 3V-to-5V data inverter chip but I’d like to avoid that one if possible.)

PowerSupply

Update 3

I’ve smoke-tested the power supply/ring combination and it’s looking good. To make things easier for this step, I’ve now setup a surrogate Raspberry Pi 3 for testing things but since I only had a leftover 4GB microSD, I was forced to use the no-desktop “Lite” Jessie version of Raspbian. But that’s now ready and I’ll likely have some time this weekend to do a basic blink test.

despicable me—themed supercomputer

I gave a talk on Tuesday to an eager group of 155 attendees at the monthly SanDiego.js meetup on the topic of “Supercomputing in JavaScript”. I had an opportunity to show the new Raspberry Pi 3 supercomputer which I’d built and took it through its paces.

I think they mostly loved the audio events for assembling the minions and sending them to bed (shutting off the remote nodes). There was just enough time to also show the obligatory “Hello, Minions!” demo program to exercise the Message Passing Interface. I received a wide variety of questions and compliments from the group. And of course afterward, everyone who owned a Raspberry Pi came over to discuss their own projects, which was cool.

Here’s the PowerPoint presentation from that talk, in case you’re interested.

e-mc2 repository with step-by-step instructions

open-source minion supercomputer

It seems like every villian these days has their own plan for world domination. So I thought I’d better get started on my own by developing an open-sourced supercomputer design using the affordable Raspberry Pi 3 computer as the basic building block.

Here, I list some of the pertinent points regarding that design.

Minion Theme

The usual master-and-slaves paradigm for most supercomputer systems is replaced here with a gru-and-minions theme. I intend to add sound events at some point in the future, given the usual “headless computer” arrangement for servers. This should help to flag certain events as they’re happening and add some fun to the project as a whole.

Modular

Each squad of four computers would be contained in such a way that it could connect together with the next set of four.  The name for each squad is E=MC2 which stands for Expandable Minion Cube Cluster.

Self-Contained

Each E=MC2 has its own internal Ethernet switch and a USB charging hub to power things inside. The first in the system also includes a shared USB-based thumbdrive for common resources as well as the code that you wish to run in parallel.

Low-Power

Everything internally is powered with a USB charger so that’s about 15W per computer (or core). The power cost then is less than half the per-core cost of a typical supercomputer for electricity, noting that your average supercomputer must also be cooled.

Low-Cost

Weighing in at less-than-$300, the squad of computers has a per-core cost of less than $75. Throughout, I’m using open-source software. I hope to further minimize costs involved by renting time on a 3D printer locally rather than purchasing one.

Mobile

Since I hope to show this project to others, the entire thing should be relatively mobile by design.

In the “known wi-fi” mode, a single power cable could be the only connection to the device, allowing a laptop to connect to it wirelessly. It would be necessary to first educate the gru node and the laptop with the wi-fi credentials, however.

In the “unknown wi-fi” mode, you would add an Ethernet cable to a laptop computer to connect to the system. It would be necessary to configure the laptop’s Ethernet adapter to match that of the internal wired network.

When the 3D printing work has been completed I envision a clear, self-contained, 6″ cube weighing in a under two pounds (minus the power cord). We’ll see how that plays out soon enough.

Technical Savvy

I’m trying to make this as easy as possible for others to repeat. I’m carefully documenting the hardware list to make this easier to source and so that the same set of 3D printing designs I use might accommodate your own attempt at this.

I do, however, assume that you would have some basic electronics experience, have some tools and a small budget to work with and that you understand your way around Linux terminal commands like ls, cp, mv, mkdir, cd, chmod, chown, sudo, apt-get install as well as how to map a drive on a network in either OS X or Windows, say.

You’d also need to know how git and how github works in order to pull the code and design files.

It will be helpful if you understand DNS, DHCP, the concept of IP addresses and how to configure these on a computer.

Documentation in Real-Time

I’ve purchased the new hardware for this, having done some work already preparing the working image for gru, the master node of the system. I’ve created a repository for everything at e-mc2, noting that there are more than one README.md file within the collection of instructions so far.

As I install the full set of software and dependencies again I will be updating the documentation so that everyone may follow along and be assured that I’ve not glossed over any of the details.

Design Hurdles

I had some problems with the first (several) attempts to power everything with a chassis-based switching power supply but the connections themselves proved to be too problematic. And given the open-sourced nature of the project I would hope that others would build their own so I deemed this attempt “too fussy”. Additionally, the “green” Ethernet switch’s tendency to lower its power demands before any nodes came on then confused this power supply which looked something like this:

  1. The “Green” Ethernet switch would power on and the switching power supply would adjust the input current and output power
  2. The switch would determine that none of the nodes had “linked in” to their Ethernet connections (yet)
  3. The switch then would go into green mode and lower its power consumption and the switching power supply then would attempt to toggle down the output power
  4. The switch and other devices wouldn’t get enough power and then would power cycle, repeating the pattern

This next iteration then changes the power system completely which so far promises to be much easier, resulting in solid connections to the devices.

Common Protocols and Code

I’ve decided to use the Message Passing Interface (MPI) for inter-node communications and the Samba software so that the coder may map a drive to the shared data/code area.

Since I like Node.js, I’ll be developing something so that this is a choice as well. The initial “Hello, Gru!” demo may likely be written in Python, however.

Plastic

I’ve found a local place that rents 3D printers. Hopefully I can get some help designing the files necessary to house all this in a clear cube.

As far as external connections/openings to the cube, there should be one rectangular cutout for the IEC320C5 three-prong power plug and a Keystone receptacle for the Ethernet connector so that it can snap into place. Also, there should be some holes adjacent to an internally-mounted speaker. Optionally, there might be openings on both sides if a fan is added. Overall, it should be a pair of clear plastic parts which snap together to form a cube.

Internally, there should be screw holes and possibly slide rails or bumpers for the Ethernet switch, the USB charging hub and the four Raspberry Pi 3 computers. There should be room for five 1′ Ethernet cables as well as five or six 1′ micro-USB cables for power. As preferences go, it should be relatively easy to open up, remove a computer and its micro-SD card for the sake of maintenance, say.

Although it should operate well without a fan, I’m trying to source a usable 5V-powered fan for the design. Since some of you will want to overclock, this is probably a good idea, I’d guess. That being the case, a cube-to-cube side orientation of venting would create a continous flow of air through all. It may be difficult for everyone to source something like a 5V-to-12V boost module but that’s probably the route to go, combining this then with an inexpensive and readily-available 12V computer fan.

Using the Last USB Port Wisely

The existing design calls for four USB cables to power the computers and a fifth to power the Ethernet switch. That leaves one last USB port to power something. Here are some options.

  1. The existing design utilizes a tiny USB thumbdrive which goes directly into the gru node and is shared by all. Instead, this could be a USB-powered external drive with faster throughput, if so desired. One USB cable would connect the drive to gru and the second would provide power over USB to the charging hub.
  2. A small speaker attached to the grunode should allow sound events to be played in conjunction with code execution. Some speakers are USB-powered and this may be an option.
  3. A USB-powered fan would be the logical choice, however, for the remaining port.

Repository

In case you missed the link above, the repository is now at e-mc2 on github. As of today, it’s probably a bit premature to start your own supercomputer from these designs. I personally would wait until I’ve got more of the MPI-based coding in place and have added the step-by-step instructions for some of the dependencies.

Update 3/27/2017

Things are looking great for the repository and the supercomputer itself. The MPI-based code is now working as well as the website for managing the minions.


server-side tone synth with node

I thought I’d work on another Raspberry Pi 3 project and this time was looking to expand my repertoire for server-side audio.  I suppose I was imagining an application in which you’d carry the Pi along with you using a USB-based power bank and plugging in your headphones, controlling the interface via your cellphone’s browser, let’s say.

Platform

Logically then this would need to be a Node.js—based webserver which could synthesize a sine wave tone with a specified frequency on both left/right audio outputs on the server itself (rather than streaming it back to the client browser).  I chose the Raspian image for the microSD card because, as a Debian modification, it’s likely the closest to the standard Ubuntu flavor of Linux which I work with the most.

Open Source

Searching the npm space I found nothing useful, to be honest. It’s rare when the open source world fails to deliver but this was one of those times. I was going to need to innovate on this one. There are many repositories out there which use the browser’s window object but that just wasn’t available here on the server. I installed module after module only to find that one of its dependencies relied upon that same pesky window object. So each time, I had to perform another npm remove --save whatever from my project to return things back to “square one”.

This research took a little longer than I’d hoped for but it turns out that Raspian out-of-box includes the Open Sound System (OSS) aspi-utils collection and its speaker-test program can be used in a terminal to generate tones from both speakers. Okay, so now how do you run terminal programs from JavaScript?

Starting External Programs From Node

It turns out that Node has an internal module called child_process which allows you to spawn (run) something you might normally invoke from the command line. It took some reading but I was able to navigate the documentation and to start a speaker-test session with the necessary arguments. One task down, (one to go).

Stopping External Programs From Node

Fortunately, Node also has an internal module process with a kill command to stop a running process.

Implementation

It takes a bit of state management: the webserver needs to remember if you’ve started a process, what its process ID is and whether or not you’ve now stopped it. A proper open source project would support systems other than Linux but this is the starting point and this wouldn’t work, say, with a Windows-based computer using the PowerShell module control, for example.

Since this wasn’t an exercise in UX design the interface couldn’t really be any simpler. It’s your basic Express-generated website with the Pug view engine and a simple pull-down list for selecting some preset frequencies. Submitting your tone choice POSTs back to the same index page; selecting the Stop button POSTs to a virtually-routed /stop routine which then redirects back to the original index page.

Source Code

Check it out if you’re curious. I’ll likely be using this same technique to create a more self-contained module for managing external programs.

github.com/OutsourcedGuru/heal