QNAP

Good Settings for QNAP

2 Comments

The last few years of using my QNAPs have been great. QNAP isn’t perfect but it is a good, quiet, Linux-ish platform for serving up files. Since some of my family are buying their own, I thought I would document some of the settings I commonly change:

  • General: I force HTTPS (its silly and unsecure not to); I tend to give QNAPs hostnames that allow for future growth (fenqnap-1, fenqnap-2), etc., Synchronize with NTP (important for making sure that a pair of QNAPs all are in sync),
  • Storage manager: When i provision a new QNAP I use thin provisioning to form a storage pool, as this lets you flexibly break out individual volumes of any size – either thin or thick will let you do snapshotting. On my volumes I always enable encryption – but DO NOT save the password (it entirely defeats the purpose of disk encryption!). Also do NOT use static single volume – its too limiting!
  • Networking: On my TS-453 Pro I have four NICs. I bond two using the ALB scheme, and use that as my default gateway. I then leave two free, since for virtualization I only had luck configuring the virtual switch on a non-trunked eth. The ALB scheme has two advantages: load balancing and redundancy. Each client will ask for files over a given interface, but the QNAP use an output eth based on a hashing scheme. Additionally, having ALB turned on means that if one link goes down, it will be removed from the hashing scheme, and the other links will be used instead.
  • Security: I allow all connections, but on “Network Access Protection” I lock multiple failed attempts out forever
  • Hardware: I disable beeps for system operations – QNAPs are a bit too beepy
  • Power: I make sure the QNAP always turns back on.
  • Notification: Configure to send alerts to your email
  • Shared Folders: I like to group things at a shared folder level – pictures in one folder, docs in another, music in another, ISOs in another, etc. It turns out that if you don’t group by the shared folder level, you can run into some funky permissions problems (specifically: I found I couldnt reliably restrict access to my photos even when “Advanced Permissions” clearly limited access – it appeared that QNAP was accessing photos via the admin user, which always has access; i filed a bug with qnap but they dont appear to have resolved it entirely). I also enable “Advanced Folder Permissions” to let me lock out individual files via setfacl/getfacl.
  • Network sharing (“Win/Mac/NFS”) – I like leaving windows (smb) and mac (AFP) enabled, but NFS just isnt secure, so i disable it.
  • FTP: Disable it – its unsecure
  • Network recycle bin: I disable it – it doesn’t fit any workflow I use, and I dont want stuff piling up in it.
  • QSync: In the past I have disabled this, but I do think it could be useful. Until recently I was pretty fine keeping all my data on the NAS. There are times, when I am away from the Internet for example, where I want to at least, say, edit my journal, then sync to the NAS later. QNAP’s QSYNC feature directly addresses this concept, even letting you resolve conflicts (say multiple people are editing a file, for example) and keep versions. QNAP stores your qsyncs inside the users home directory on the NAS. Even though this feature is useful, I think I still prefer my own thing: 1) nightly snapshots on the QNAP, and 2) rsyncs in crontabs. Why? First of all, QSync only runs on mac/windows, so Linux is SOL. Second, the QSync client is very limiting: it only lets you specify a single location where you put all your files. That might work, except even if I symlink in other locations it doesn’t do the right thing. Nothwithstanding my reservations, I think QSync is great for single-folder to single-folder stuff. In my case Ill stick to rsync and crontab.
  • Station manager: I disable the music station, as the iTunes server is the only real way I would stream music from the QNAP.
  • Multimedia management: I disable indexing images in my document folder. The idea is: I’d like one location where even images, such as sensitive document scans, aren’t indexed for general viewing
  • VPN Server: I enable this, even though I don’t use it, because QNAP appears to do crazy things when running your own openvpn client unless there is already some VPN service enabled.
  • Antivirus: I do daily scans
  • *: Pretty much everything else gets disabled

On top of this I configure my own key-based (passwordless) VPN, and then setup individual backup jobs in the backup station. I describe how to set up password-less VPN in a previous post. Each job connects to my backup QNAP over the VPN. I enable encryption, and also have the job apply custom permissions (be sure Advanced Permissions is enabled on the other QNAP!). I have each job auto-sync on a schedule.

Note: I don’t use RTRR, even though it seems cool, because it doesn’t fit my workflow – i don’t want the QNAP auto-sycning live, as it would eat up too much bandwidth – im ok with a nightly sync. Plus I have no idea what RTRR uses as a protocol – and I am doubtful whatever it is really is superior to rsync (not that I am incredulous, but if it is QNAP sure is keeping it a secret).

Note2: On the destination QNAP I go into the Backup station and enable the rsync server – but only the middle checkbox (you dont need to enable the one that makes you enter a username and password). Once enabled, you can create a user, or just use admin, then on the source QNAP just use that users credentials to enable the rsync.

Note3: I don’t use myqnapcloud. It is a nice service, but since I have my own VPN I can access my QNAPs just as if I were at home.

Note4: In terms of extra apps I get by with simply installing the HDStation and the CodexPack to enable HW-accelerated transcoding. I also use virtualization station so I can run a few “real” linux VMs on the QNAP. I’ve been very interested in container station, but as yet havent got it to work.

Note5: Be sure you run nessus, or nmap, to get a good profile of any vulerabilities on your QNAP. I found a few ports, like 631, that absolutely did not need to be open; in some cases I found a service that was configured but that I didn’t need, so i shut it down. Sadly in the ipp (631) case, I could find no way to shut it down.

Note6: If you want to do your own version of qsync, I just setup password-less logins from all the sync “sources”, then create a directory (or set of directories) I want to sync. I wrote a single script to sync these folders, something like:

#!/bin/bash
# Backup docs
rsync -auvhP /local/path/syncToNas/* admin@qnap1:/share/CE_CACHEDEV2_DATA/DocShare/mac
# Backup passwords
rsync -auvhP /local/path/.password-store admin@qnap1:/share/CE_CACHEDEV2_DATA/DocShare/passwords/

You can put this script, call it backupMac.sh (or whatever), in your /local/path/syncToNas to be sure your script, and your local files, are synced. Your crontab would then just be something like:

* * * * * /local/path/syncToNas/backupMac.sh 2>&1 > /dev/null

Of course this only works on Mac and Linux; for windows maybe you go ahead and use qsync.

Web Programming

Learning Log: React.js and HTML5 video

Leave a comment

Carson’s Web Development Maxim: If you don’t like any of the web toolkits that are available just wait a few months: there will be ten new ones waiting for you.

I like, and still use, Angular, but I wanted to try something new. Enter React: a framework from the Facebook side of the fence. What better way to learn than with a project! I decided to make an app to help my kids clean up.

The concept around the application is to track the kids progress as they pick up toys. Specifically, they are asked to pick up six toys at a time, take a picture of each toy, and put the toy away. Thankfully I had an old Nexus 7 tablet laying around that nobody was using. Also, chrome support for HTML5 video lets a web app gain access to your tablet. Looks like I am set!

After reading through the “comments” tutorial for React I whipped up the following (shown running on the Nexus 7):

Screenshot_20151229-110001

The user interaction is designed as follows: each kid has a row for their “pickup tiles.” For each toy, they click a new tile and the app takes a picture. They can retake on a square as many times as they want. When they get all six squares a tantilizing reward appears: an animated gif of their choosing (yes, yes — the 90s are calling and they want their webpage back!) This is shown below:

Screenshot_20151229-110407

This app was interesting from a HCI perspective. The following observations were made:

  • Kids had a hard time using the tablet to “click” on a tile. I think their little fingers don’t register too well on a tablet.
  • There was no way to preview: maybe clicking should have shown a live preview which snapped the picture after a 2-second delay.
  • The kids were more interested in taking pictures of toys than putting them away. This resulted in toys being piled up near the tablet, rather than being put away, as kids ran around finding new toys to take pictures of.
  • Pickup actually took longer than without the app. Even when the kids put things away they took a lot longer staring at the screen.

Originally I was going to back this app with a server-side component that would store all pictures. The thought being that it would be fun to see the silly things the kids picked up. In the end I decided to skip this because of my observations from some trial runs, which basically indicates that the app makes pickup worse. I’ll just stick the traditional approach of motivating kids (e.g. “Pick up your toys or I’ll give them to Chuck at work”).

From a programming perspective here are a few observations:

  • React is awesome. I was amazed at Angular’s novel approach that effectively extended html to allow you to embed display logic into your markup. My mind was blown again with JSX, which essentially goes the opposite route, letting you embed markup naturally inside your code. I liked the way you could easily form components and interact with their data models.
  • Also in terms of html 5 video in Chrome: this worked perfectly. Only annoyance is that you must use https to get at the camera, which doesn’t seem necessary on a local net. Also the aspect ratio on my Nexus 7 is weird when in portrait view.
  • There is a ton of wasted space on chrome on the tablet. I couldn’t find any currently-supported method for making a web app full screen. There had been support in the past, but when i looked in chrome’s settings it appears the options are no longer available. Wouldn’t it be nice to see your app in full screen, especially on such a space-constrained device?

The source code for my app is here

Containers

Containers for Fun and Profit

Leave a comment

With all the whirl about containers I decided I could wait no longer to join the fray. Here is a log of some of the things I learned.

Basic test, e.g. what the heck is this?

I spun up a cent7 vm several months back. Apparently my repos were a tad old and things didnt work until i did a yum update first, then reboot. Then I could systemctl enable docker, systemctl start docker

Note you MUST sudo in for docker commands to work

Also note: cent7 has docker in it natively – no need to wget install it (unless you want to use some of the new features like the networking module)

The following is your smoke test:

sudo docker run hello-world

Test 2: Expanded sample

This worked flawlessly…

Test 3: Interactive sample

Ditto, worked perfectly

Test 4: Do my own thing

I made a “looper.py” app with the following code:

#!/usr/bin/python
import time
while True:
	print "Hi! ", time.time()
	time.sleep(1)

I then made the following Dockerfile

FROM docker.io/centos:latest
RUN yum install python -y
COPY looper.py /
CMD /looper.py

And create the image with 

	docker build -t looper .

When i ran the build one of the things I noticed is that python was already installed in the centos image. I modified the Dockerfile by removing the RUN line, and one cool thing is that when i re-ran the build command, the python install layer was automatically removed, and everything else was basically a noop. In other words docker appears to do a good job at being efficient.

I then ran my looper:

	docker run looper

Nothing happened… So i thought…and though… and eventually decided to try and attach. Ubeknowest to me, by doing docker run i WAS attached, but nonetheless I learned a few things:

  • To attach you need your container id
  • To get your container id you run “docker ps”

Once I did a docker attach to my container id, i saw nothing, still. I did a Ctrl-C and viola, my looper output appeared! I suspected buffering, which turned out to be the case. I modified looper as follows:

#!/usr/bin/python
import time
import sys
while True:
	print "Hi! ", time.time()
	sys.stdout.flush()
	time.sleep(1)

Then rebuilt, and re-ran, and it all worked.

Note that this only runs the command in the foreground. To run it in the background:

	docker run -d looper

You can then docker ps, find the cid, docker attach to it. But… you cannot detach (without sending a SIGKILL)! The docks say Ctrl-P + Ctrl+Q will detach, but this appears to only work if you use the following command when running it:

	docker run -tdi looper

Where t means create a tty, and i means “keep stdin open even if not attached”. This works well.

Note that each time i make changes to looper, when i rebuild it takes at most 20 seconds.. if no changes, docker takes milliseconds…

Test 5: layers

What if the container modifies a file?

I modified looper.py to write to stdout and a file:

#!/usr/bin/python
import time
import sys
while True:
	msg = "Hi! " + str(time.time())
	print msg
	with open('myfile','a') as f:
		f.write(msg + '\n')
	sys.stdout.flush()
	time.sleep(1)

For fun i created a file named “myfile”, then build the image, then ran the container. When it runs i can do a docker diff:

# docker diff f04849645523
A /myfile

And to be clear, this means the file was added in the image. Docker wont let the app reach into my own version of “myfile”.

What if i want to see the file? In older versions of docker, apparently you had a few options, such as running ssh, or making a snapshot, but now its easy:

	docker exec -t -i <cid> /bin/bash

You can then just cat the file, etc. If you actually want to copy the files out,
you can export the whole filesystem (docker export ) as a tar, but this seems nuts. If you just want a single file, use docker cp:

docker cp <cid>:<src> <dest>

Test 6: CPU limit

You can do a couple things:

1) Limit the share of cpu usage across multiple containers. This is done by specifying a relative weighting (with -c)

2) Pin the process to certain cpus with —cpuset-cpus=

I havent been able to find an equivalent to the simple “limit to N processors” idea on virtual machines. The weighting is fairly close.

QNAP

Migrating QNAP Static Vol to Storage Pool

3 Comments

WARNING: This is risky. Just assume that you would only try this if you were ok losing your data.

When I got my first QNAP almost two years ago it didn’t support storage pools (or if it did I was oblivious). One of the advantage of storage pools is that they enable QNAPs snapshot replica feature. The trick is that you cannot migrate static volumes to storage pools, or so so says QNAP.

But it turns out there is a way to do this. Say you have a RAIDed static volume. There are at least two disks in such a volume, and you can withstand the loss of a single drive. You can “migrate” your data as follows:

  1. Shutdown NAS
  2. Remove all but one drive
  3. Power on NAS and wait for it to boot up. If you go to it (using QFinder for example) it will indicate that there are no disks.
  4. Insert a single drive (henceforth dubbed the “pool drive”)
  5. Choose “Restore to factory”
  6. Once it boots, delete the volume
  7. Use the deleted volume to create a storage pool and a new volume (I chose thin, because “why not”)
  8. Once the snapshot pool and vols are created, shutdown the NAS
  9. Remove the pool drive and insert the other drives
  10. Once the nas starts up restore to factory settings again.
  11. Once the factory reset is complete insert the pool drive again.
  12. You can now create shared folders on the storage pool, then rsync from the old static vol to the new storage pool.

The final step, once this is done (assuming you didnt lose any disks along the way!) you can expand the storage pool using the static vols, and viola, you’re back in action!

DIY Cell Phone

GSM Phone Tracking Methods

Leave a comment

I decided to conduct a few tests with my fona808.

  • Battery. The at+cbc command give you the current charge mode, percent charged, and millivolts. I found on the adafruit fona808 that the charge mode indicator always gave status 0 (“not charging”) even when charged.
  • GSM Location. This is given by the at+cipgsmloc. It spits out lat and lon. I don’t know where it is getting the data from, but based on the plots I made the location it gives is an estimate based on your location to the nearest cell tower.
  • GPS. Given by at+cgnsinf. I found this to be spot-on, always accurate, even with my pea-sized GPS antenna!
  • DIY cell tower triangulation. If you put the fona into ENG mode (“AT+CENG=3”), it will give you the MNC, MCC, LAC, and CellId for the towers around you (i usually got six reports per “AT+CENG?” query). You can then use a site like cellphonetrackers.org to turn the tower info into a lat/lon coordinate. The +CENG messages also given a power level which you can use, in cojunction with the cell tower lat/lon coordinates, to perform the triangulation. The method of doing this is described elsewhere, but basically each power level becomes a weight, w_i = rx_i / (rx_0+rx_1…rx_n), which is multiplied by the lat/lon of the corresponding cell tower item. You then just add up the weighted lat/lons, and viola!

      • My results?

      • I found that i lost about 10% power on my 1 hour trip. This seems terrible – barely 10 hours per charge, extrapolating.
      • The GPS info was perfect. It took about 2 min to acquire
      • The GSM location was too course, but was actually more clean than my DIY location
      • DIY location seems crumby. Im not sure what would fix this. Possible things to look at: 1) Filtering. Maybe I could throw out the lowest power rating, or smooth out the locations somehow. 2) See if there are better cell tower dbs. As far as i know, cell tower info isnt public, so any db is most likely based on reported, possibly inaccurate, values

      Here’s a map with plots of the three localization methods: GPS (magenta), GSM (green), and DIY Triangulation (yellow).

      gsmtrack

DIY Cell Phone

My favorite FONA commands

Leave a comment

These are more like “the commands i have found useful as of present.” I bear no real affinity for them, except that I do appreciate the data they yield.

AT+COPS? Ensure you are connected to the network (it gives “+COPS: 0” if you are not)
at+ccid Get the SIM number; you need this for activation
AT+CMGF=1 This sets us into text mode. I haven’t used the other mode (PDU) yet.
at+sapbr=3,1,”contype”,”gprs” You can set the connection type to GPRS (data) or CSD (circuit switched) – i think this is why you can call or do data on gsm networks, but not both (at the same time).
at+sapbr=3,1,”apn”,”wholesale” Until you set your access point name, you might not be able to do things like geolocate (based on cell towers) or do data stuff. TING’s APN is “wholesale” – im sure it is different for every provider.
at+sapbr=1,1 Open up your bearer. . . sounds good, but im not entirely sure what that means.
at+cipgsmloc=1,1 This gives you your lat/lon. Note that on my old T-Mobile sim card, which had no data plan, i got nothing back for some reason; apparently you have to have a data plan to get this info?

For SMS (which i used only briefly) I found the following useful:

at+cmgl=”all” I didn’t realize this, but all text messages really are stored somewhere in the providers network, at least until you do something with them (makes sense) – i just found it interesting that I transplated my SIM card from my tmobile device to my FONA, and could see text messages from years ago.
at+cmgs=”180188xyzwl” This is of course how you send a text. You press enter, after typing the phone number (“1801..” – notice the leading “1”, since im in the USA; not sure if this is needed, but it works with it). When you are done you must hit ; if you hit it cancels the message!
at+cmgr= This is your way of reading a text message

For GPS I used the following (note I have v2 of the FONA808 – the commands are different for v1)

AT+CGNSPWR=1 The device starts with the GPS off. So i turn it on, cause i want it.
AT+CGNSINF Gives you a crudload of gps-related info, including lat-lon, altitude, utc, etc. (see table 2-2 in the “SIM800 GNSS Application Note”)
DIY Cell Phone

DIY Phone

Leave a comment

I was recently granted an adafruit shopping spree by my wife. I chose a FONA808 and some Trinket microcontrollers (literally they are about the size of my thumbnail!).

Im still learning the ins and outs of the FONA, but it is amazing how much momentum you can gain by simply following the instructions listed in the adafruit tutorial. Since I spent a couple hours due to my failure to read I thought I would pass on for some other soul, like myself, the importance of simply doing what lady ada says. Namely: 1) DO connect your power source (battery, usb +5, whatever) to VIO, and 2) if you don’t see the blue light shining hold down the little button on the module for two seconds (or just tie KEY to ground). These two things will work wonders.

Oh, and for those using OS X, I found much success by using the USB serial cable available on adafruit. I just downloaded the drivers from the product page, downloaded coolterm, and away i went. The autobaud on the fona works marvellously, as witnessed by it echoing back after I typed “at” a few times.

Backups

If you had to do DVD backups from windows

2 Comments

Say you want to let your grandmother backup her DVDs using her clunky old windows laptop. You want a windows solution that 1) rips the DVDs, 2) copies them to a backup, 3) makes them available for viewing from wherever (e.g. accessible over the internet).

The solution I came up with is as follows:

  • Download cygwin and install rsync. I hadn’t used cygwin for a long time, but basically you download either the 32 or 64 bit version here, then filter for rsync and select it. Cygwin seems to do a good job of only pulling what is necessary.
  • Download HandbrakeCLI. You dont need the full Handbrake installer, just the CLI. You should be able to google for it, but i found it here
  • libdvdcss. This is all over the internet. Just google and ye shall find. You need this to be able to decrypt most DVDs. Just place the DVD dll inside your handbrake folder to enable decss capability.
  • Download VLC. I felt like ripping my DVDs with chapter markers. The best format for supporting this kind of thing seems to be matroska (mkv). VLC natively supports mkv, which is handy if youre viewing these DVDs locally

I then automated the whole thing with the following script:

@echo off
SETLOCAL ENABLEDELAYEDEXPANSION
set /p dvdname= Enter DVD Name (no spaces):
set basepath=<localScratchDir>%dvdname%.mkv
set remotedir=<remoteBackupPathDir>/%dvdname%.mkv
FOR /F "tokens=* USEBACKQ" %%F IN (`c:\cygwin64\bin\cygpath %basepath%`) DO (
SET basepathunix=%%F
)
echo Ripping to %basepath%
HandBrakeCLI --main-feature -m -i d: -o %basepath% -e x264 -q 20 -B 160
echo Copying %basepathunix% to backup
c:\cygwin64\bin\rsync.exe -e "/usr/bin/ssh -p <remotePort>" -auvhP %basepathunix%  <remoteUser>@<remoteSite>:%remotedir%
echo.
echo.

if "%errorlevel%"=="0" (
        echo Completed successfully!
)
if not "%errorlevel%"=="0" (
        echo DVD backup did NOT complete successfully!
)
echo Removing %basepath%
del %basepath%

set /p dummy=Press enter to finish

Note some assumptions about this script:

  • You are running 64bit cygwin. If not change from c:\cygwin64
  • You want to push to somewhere offsite. You can just skip the rsync step if you want to go local.
  • Note that I used a custom port in this example; you can remove the -e switch and its argument if you’re using a standard port.
  • This example assumes you have set up ssh keys properly. Meaning you ssh-keygen on your windows box (from your cygwin shell!) then copy your key to the dest machine (perhaps using ssh-copy-id). I don’t allow password authentication, so somtimes its easiest to just ssh into the remote host somewhere that has access, then teamviewer/rdesktop/whatever into your windows box to copy the ~/.ssh/id_rsa.pub file into the ~/.ssh/authorized_keys file.
  • I don’t care about extra content, so im only ripping the main feature.

As for legally accessing your DVDs remotely: you can use one of any methods. I have found PLEX to work well.

Frequent Flyer

Airlines / Frequent Flyer Programs: The good, the so-so, and the ugly

Leave a comment

An earlier blog post detailed heroic feats of airline reward redemption: leaping across multiple rewards programs, bounding over unaccompanied minor restrictions, and otherwise gaining cross-country access for peanuts!

And then everything was shattered when I had to change my tickets.  This let me see how well each of the rewards programs faired in handling booking changes.  This leads me to my list of good, so-so, and ugly:

Let’s start with the ugly: Delta.  I think its kinda neat that I booked $70 flights from LAX to SLC.  Then when I wanted to change the flights they were willing to do so for $200 per ticket.  I’d rather pay a bit more for Southwest and get better customer service.

Now the so-so: american airlines earns my so-so category.  They didnt really provide any way to change my existing flight, much like delta. However they have otherwise great service, and some of the best reward travel coverage available; I can always find an AA flight, even through avios.

Which brings me to my winners, the good guys: Southwest and Avios!  I must first say southwest is my all around favorite: TWO free checked bags, NO fees when changing flights.  Awesome.

And Avios deserves a good apology from this author.  I was able to use only 13500 points to book a one-way flight for three of my family members from LAX to SLC.  That same thing would have taken over 30k points on something like southwest or aa.  It turns out that if you aren’t bounding from coast to coast avios is a powerful tool to have in your points chest.  Its fantastic for those short-haul trips.

There ye have it.

Uncategorized

A Tasty Apple Pie Recipe

Leave a comment

Having a bushel or two of apples on hand, I decided to make an apple pie the whole family could enjoy.  I borrowed two recipes.  For posterity i have included the actual recipe in addition to the links.

For the apple pie guts I used Apple Pie by Grandma Ople:

1 recipe pastry for a 9 inch double crust pie
1/2 cup unsalted butter
3 tablespoons all-purpose flour
1/4 cup water

1/2 cup white sugar
1/2 cup packed brown sugar
8 Granny Smith apples – peeled, cored and sliced
Directions
Preheat oven to 425 degrees F (220 degrees C). Melt the butter in a saucepan. Stir in flour to form a paste. Add water, white sugar and brown sugar, and bring to a boil. Reduce temperature and let simmer.
Place the bottom crust in your pan. Fill with apples, mounded slightly. Cover with a lattice work crust. Gently pour the sugar and butter liquid over the crust. Pour slowly so that it does not run off.
Bake 15 minutes in the preheated oven. Reduce the temperature to 350 degrees F (175 degrees C). Continue baking for 35 to 45 minutes, until apples are soft.

and for the crust I used French Pastry Recipe:

Ingredients
3 cups all-purpose flour
1 1/2 teaspoons salt
3 tablespoons white sugar
1 cup shortening

1 egg
1 teaspoon distilled white vinegar
5 tablespoons water
Directions
In a large mixing bowl, combine flour, salt, and sugar. Mix well, then cut in shortening until mixture resembles coarse meal.
In a small bowl, combine egg, vinegar, and 4 tablespoons of water. Whisk together, then add gradually to flour mixture, stirring with a fork. Mix until dough forms a ball. Add one more tablespoon of water if necessary.
Allow dough to rest in refrigerator 10 minutes before rolling out.

The results we’re fantabulous:

P1060846