I purchased a couple raspberry pi 3s. The major draw is that you no longer need an external wifi adapter – yet the cost is the same as the older pis.
The very first thing I noticed is that ssh traffic was stuttery. This cleared up after performing an apt-get update/upgrade. Everything seemed silky-smooth thereafter.
As nice as the photo station is inside of the QNAP, sometimes you just want to do your own thing with your photos and videos. But what if they are organized into albums on the QNAP? Couldn’t you just pull down the photos/videos in the album? Or, better, get local paths on your QNAP network mounts? All this, and more, is possible with the following script:
#!/usr/bin/python
import shlex
import subprocess
import os
parms = {}
parms['host'] = "qnaphostname"
parms['login'] = "admin"
parms['albumName'] ="Album1"
localBase = '/Volumes'
cmdBase = """ssh %(login)s@%(host)s "/usr/local/mariadb/bin/mysql -u read --password=read s01 -B -S /tmp/mysql_mediadb.sock -e 'select cFileName, cFullPath FROM pictureAlbumTable picAlbum JOIN pictureAlbumMapping picMap ON picMap.iPhotoAlbumId = picAlbum.iPhotoAlbumId JOIN %(mediaTable)s pic ON pic.%(mediaColumn)s = picMap.iMediaId JOIN dirTable ON dirTable.iDirId = pic.iDirId WHERE picAlbum.cAlbumTitle = \\"%(albumName)s\\" and type = %(type)s; ' " """
tables = []
tables.append({'mediaTable':'pictureTable','mediaColumn':'iPictureId','type':'1'} )
tables.append({'mediaTable':'videoTable','mediaColumn':'iVideoId','type':'2'} )
allFilenames = set()
for table in tables:
parms.update(table)
cmd = cmdBase % parms
print cmd
res = subprocess.check_output(cmd, shell=True)
# Skip the first row, which has column headers
hits = res.split("\n")[1:]
print "Found %s hits" % len(hits)
for f in hits:
parts = f.split("\t")
if len(parts) != 2:
continue
filename = os.path.join(localBase,parts[1],parts[0])
allFilenames.add(filename)
open(parms['albumName'],'w').write("\n".join(allFilenames))
This grabs video and photos using ssh to run the mysql query remotely. QNAP references videos and photos from a very similar set of tables in a mysql database (called s01 on my box). The username and password for this database is read/read. The -B option makes the results tab-delimited, and thus more parseable.
Of course this isn’t super polished or anything; just modify the host and albumName parameters and you should be fine. That is, assuming you’re on OS X. If you are not, modify localBase to point to your mount location for the QNAP shared folder.
Update: I missed this originally, but you also have to pass in the type for each of the two queries. If you don’t do this you invariably get more videos and pictures than are actually in the album. The query is now updated above.
Open source software rules the internet. The majority of website are supported in some form by open source: be it linux, apache, or some open source library or language.
Surely, then, we could find an open source alternative to social media websites like Facebook, instagram, and Flickr… right?
Here are my requirements:
Here are the options I found:
Given the state of things, I’d say Lychee is my first pick, were it not for missing multi-user and comment features. For now I’m sticking with Piwigo.
The last few years of using my QNAPs have been great. QNAP isn’t perfect but it is a good, quiet, Linux-ish platform for serving up files. Since some of my family are buying their own, I thought I would document some of the settings I commonly change:
On top of this I configure my own key-based (passwordless) VPN, and then setup individual backup jobs in the backup station. I describe how to set up password-less VPN in a previous post. Each job connects to my backup QNAP over the VPN. I enable encryption, and also have the job apply custom permissions (be sure Advanced Permissions is enabled on the other QNAP!). I have each job auto-sync on a schedule.
Note: I don’t use RTRR, even though it seems cool, because it doesn’t fit my workflow – i don’t want the QNAP auto-sycning live, as it would eat up too much bandwidth – im ok with a nightly sync. Plus I have no idea what RTRR uses as a protocol – and I am doubtful whatever it is really is superior to rsync (not that I am incredulous, but if it is QNAP sure is keeping it a secret).
Note2: On the destination QNAP I go into the Backup station and enable the rsync server – but only the middle checkbox (you dont need to enable the one that makes you enter a username and password). Once enabled, you can create a user, or just use admin, then on the source QNAP just use that users credentials to enable the rsync.
Note3: I don’t use myqnapcloud. It is a nice service, but since I have my own VPN I can access my QNAPs just as if I were at home.
Note4: In terms of extra apps I get by with simply installing the HDStation and the CodexPack to enable HW-accelerated transcoding. I also use virtualization station so I can run a few “real” linux VMs on the QNAP. I’ve been very interested in container station, but as yet havent got it to work.
Note5: Be sure you run nessus, or nmap, to get a good profile of any vulerabilities on your QNAP. I found a few ports, like 631, that absolutely did not need to be open; in some cases I found a service that was configured but that I didn’t need, so i shut it down. Sadly in the ipp (631) case, I could find no way to shut it down.
Note6: If you want to do your own version of qsync, I just setup password-less logins from all the sync “sources”, then create a directory (or set of directories) I want to sync. I wrote a single script to sync these folders, something like:
#!/bin/bash # Backup docs rsync -auvhP /local/path/syncToNas/* admin@qnap1:/share/CE_CACHEDEV2_DATA/DocShare/mac # Backup passwords rsync -auvhP /local/path/.password-store admin@qnap1:/share/CE_CACHEDEV2_DATA/DocShare/passwords/
You can put this script, call it backupMac.sh (or whatever), in your /local/path/syncToNas to be sure your script, and your local files, are synced. Your crontab would then just be something like:
* * * * * /local/path/syncToNas/backupMac.sh 2>&1 > /dev/null
Of course this only works on Mac and Linux; for windows maybe you go ahead and use qsync.
Carson’s Web Development Maxim: If you don’t like any of the web toolkits that are available just wait a few months: there will be ten new ones waiting for you.
I like, and still use, Angular, but I wanted to try something new. Enter React: a framework from the Facebook side of the fence. What better way to learn than with a project! I decided to make an app to help my kids clean up.
The concept around the application is to track the kids progress as they pick up toys. Specifically, they are asked to pick up six toys at a time, take a picture of each toy, and put the toy away. Thankfully I had an old Nexus 7 tablet laying around that nobody was using. Also, chrome support for HTML5 video lets a web app gain access to your tablet. Looks like I am set!
After reading through the “comments” tutorial for React I whipped up the following (shown running on the Nexus 7):
The user interaction is designed as follows: each kid has a row for their “pickup tiles.” For each toy, they click a new tile and the app takes a picture. They can retake on a square as many times as they want. When they get all six squares a tantilizing reward appears: an animated gif of their choosing (yes, yes — the 90s are calling and they want their webpage back!) This is shown below:
This app was interesting from a HCI perspective. The following observations were made:
Originally I was going to back this app with a server-side component that would store all pictures. The thought being that it would be fun to see the silly things the kids picked up. In the end I decided to skip this because of my observations from some trial runs, which basically indicates that the app makes pickup worse. I’ll just stick the traditional approach of motivating kids (e.g. “Pick up your toys or I’ll give them to Chuck at work”).
From a programming perspective here are a few observations:
The source code for my app is here
With all the whirl about containers I decided I could wait no longer to join the fray. Here is a log of some of the things I learned.
Basic test, e.g. what the heck is this?
I spun up a cent7 vm several months back. Apparently my repos were a tad old and things didnt work until i did a yum update first, then reboot. Then I could systemctl enable docker, systemctl start docker
Note you MUST sudo in for docker commands to work
Also note: cent7 has docker in it natively – no need to wget install it (unless you want to use some of the new features like the networking module)
The following is your smoke test:
sudo docker run hello-world
This worked flawlessly…
Ditto, worked perfectly
Test 4: Do my own thing
I made a “looper.py” app with the following code:
#!/usr/bin/python import time while True: print "Hi! ", time.time() time.sleep(1)
I then made the following Dockerfile
FROM docker.io/centos:latest RUN yum install python -y COPY looper.py / CMD /looper.py
And create the image with
docker build -t looper .
When i ran the build one of the things I noticed is that python was already installed in the centos image. I modified the Dockerfile by removing the RUN line, and one cool thing is that when i re-ran the build command, the python install layer was automatically removed, and everything else was basically a noop. In other words docker appears to do a good job at being efficient.
I then ran my looper:
docker run looper
Nothing happened… So i thought…and though… and eventually decided to try and attach. Ubeknowest to me, by doing docker run i WAS attached, but nonetheless I learned a few things:
Once I did a docker attach to my container id, i saw nothing, still. I did a Ctrl-C and viola, my looper output appeared! I suspected buffering, which turned out to be the case. I modified looper as follows:
#!/usr/bin/python import time import sys while True: print "Hi! ", time.time() sys.stdout.flush() time.sleep(1)
Then rebuilt, and re-ran, and it all worked.
Note that this only runs the command in the foreground. To run it in the background:
docker run -d looper
You can then docker ps, find the cid, docker attach to it. But… you cannot detach (without sending a SIGKILL)! The docks say Ctrl-P + Ctrl+Q will detach, but this appears to only work if you use the following command when running it:
docker run -tdi looper
Where t means create a tty, and i means “keep stdin open even if not attached”. This works well.
Note that each time i make changes to looper, when i rebuild it takes at most 20 seconds.. if no changes, docker takes milliseconds…
Test 5: layers
What if the container modifies a file?
I modified looper.py to write to stdout and a file:
#!/usr/bin/python
import time
import sys
while True:
msg = "Hi! " + str(time.time())
print msg
with open('myfile','a') as f:
f.write(msg + '\n')
sys.stdout.flush()
time.sleep(1)
For fun i created a file named “myfile”, then build the image, then ran the container. When it runs i can do a docker diff:
# docker diff f04849645523 A /myfile
And to be clear, this means the file was added in the image. Docker wont let the app reach into my own version of “myfile”.
What if i want to see the file? In older versions of docker, apparently you had a few options, such as running ssh, or making a snapshot, but now its easy:
docker exec -t -i <cid> /bin/bash
You can then just cat the file, etc. If you actually want to copy the files out,
you can export the whole filesystem (docker export
docker cp <cid>:<src> <dest>
Test 6: CPU limit
You can do a couple things:
1) Limit the share of cpu usage across multiple containers. This is done by specifying a relative weighting (with -c)
2) Pin the process to certain cpus with —cpuset-cpus=
I havent been able to find an equivalent to the simple “limit to N processors” idea on virtual machines. The weighting is fairly close.
WARNING: This is risky. Just assume that you would only try this if you were ok losing your data.
When I got my first QNAP almost two years ago it didn’t support storage pools (or if it did I was oblivious). One of the advantage of storage pools is that they enable QNAPs snapshot replica feature. The trick is that you cannot migrate static volumes to storage pools, or so so says QNAP.
But it turns out there is a way to do this. Say you have a RAIDed static volume. There are at least two disks in such a volume, and you can withstand the loss of a single drive. You can “migrate” your data as follows:
The final step, once this is done (assuming you didnt lose any disks along the way!) you can expand the storage pool using the static vols, and viola, you’re back in action!
I decided to conduct a few tests with my fona808.
My results?
Here’s a map with plots of the three localization methods: GPS (magenta), GSM (green), and DIY Triangulation (yellow).
These are more like “the commands i have found useful as of present.” I bear no real affinity for them, except that I do appreciate the data they yield.
| AT+COPS? | Ensure you are connected to the network (it gives “+COPS: 0” if you are not) |
| at+ccid | Get the SIM number; you need this for activation |
| AT+CMGF=1 | This sets us into text mode. I haven’t used the other mode (PDU) yet. |
| at+sapbr=3,1,”contype”,”gprs” | You can set the connection type to GPRS (data) or CSD (circuit switched) – i think this is why you can call or do data on gsm networks, but not both (at the same time). |
| at+sapbr=3,1,”apn”,”wholesale” | Until you set your access point name, you might not be able to do things like geolocate (based on cell towers) or do data stuff. TING’s APN is “wholesale” – im sure it is different for every provider. |
| at+sapbr=1,1 | Open up your bearer. . . sounds good, but im not entirely sure what that means. |
| at+cipgsmloc=1,1 | This gives you your lat/lon. Note that on my old T-Mobile sim card, which had no data plan, i got nothing back for some reason; apparently you have to have a data plan to get this info? |
For SMS (which i used only briefly) I found the following useful:
| at+cmgl=”all” | I didn’t realize this, but all text messages really are stored somewhere in the providers network, at least until you do something with them (makes sense) – i just found it interesting that I transplated my SIM card from my tmobile device to my FONA, and could see text messages from years ago. |
| at+cmgs=”180188xyzwl” | This is of course how you send a text. You press enter, after typing the phone number (“1801..” – notice the leading “1”, since im in the USA; not sure if this is needed, but it works with it). When you are done you must hit |
at+cmgr=| This is your way of reading a text message |
|
For GPS I used the following (note I have v2 of the FONA808 – the commands are different for v1)
| AT+CGNSPWR=1 | The device starts with the GPS off. So i turn it on, cause i want it. |
| AT+CGNSINF | Gives you a crudload of gps-related info, including lat-lon, altitude, utc, etc. (see table 2-2 in the “SIM800 GNSS Application Note”) |
I was recently granted an adafruit shopping spree by my wife. I chose a FONA808 and some Trinket microcontrollers (literally they are about the size of my thumbnail!).
Im still learning the ins and outs of the FONA, but it is amazing how much momentum you can gain by simply following the instructions listed in the adafruit tutorial. Since I spent a couple hours due to my failure to read I thought I would pass on for some other soul, like myself, the importance of simply doing what lady ada says. Namely: 1) DO connect your power source (battery, usb +5, whatever) to VIO, and 2) if you don’t see the blue light shining hold down the little button on the module for two seconds (or just tie KEY to ground). These two things will work wonders.
Oh, and for those using OS X, I found much success by using the USB serial cable available on adafruit. I just downloaded the drivers from the product page, downloaded coolterm, and away i went. The autobaud on the fona works marvellously, as witnessed by it echoing back after I typed “at” a few times.