Uncategorized

The Wonder Of Open Source And Web Development

Leave a comment

I am not a web developer. Notwithstanding this fact I have been doing web development almost every year for the past two decades. This has helped me see a recurring trend: every year I find that all the things I hated about web development have been replaced by new and exciting things. These exciting things keep their glittery shine for about six months, tarnish, wither, and become hated. I then retire from web development forever (meaning: until the following year.)

Currently I am in the honeymoon phase with React. This is after a brief love affair with Angular. I think my relationship with React will last longer because it is just so much cooler than Angular… (nevermind when my ex says “that’s what you said about me after you ditched jQuery!” Everyone knows its impossible to ditch jQuery)

The other thing of note is how many stinkin’ development tools are born every year. I was just brushing up on open source ecommerce offerings last week and stumbled upon a dozen or so. Widdling this down by filtering out inactive projects (based on github commit graphs), I found a few that seemed like good candidates. Of course this led me to some of the underlying frameworks supporting these products: composer (php – ihk; I loathe anything relating to php), meteor, electrode (from walmart labs of all places!), hapi.js, etc. What are all these heretofore-unknown-but-now-pivitol-groundbreaking frameworks?

It never ceases to amaze me how much time is spent on doing, then re-doing frameworks and toolkits. Could we solve the worlds shortage of developer problem by passing some law to limit needless duplication of efforts in creating frameworks? Maybe if we linked excess framework creation to carbon emissions Bernie sanders could jump in and help.

Perhaps toolkits and frameworks are the only fitting artifacts that a developer can leave behind for posterity. All the application code written is usually for a company, and therefore kept closed source, will likely never see the light of day. What does any good developer do besides create abstractions?

As an aside I also found it amusing to see a similar trend with amazon – no longer just s3 and ec2, they have all kinds of petabyte scale storage (including one offering that involves driving a semi truck to your business to extract up to 100 petabytes of data), lambda, snowball, blah blah, etc. Not open source, but in the same vein of neverending innovation…

Food

Oh Blue Apron!

Leave a comment

Blueapron is just plain awesome. Even if you don’t pay for it – the recipes are free! Here are a few of my all time favorites:

  • Roasted Cauliflower Meunière – Oh browned butter and lemon juice (aka Meunière), delicious is thy name! It is hard to describe how well the flavors agree in this dish. Wonderful kale and roasted cauliflower, along with this sauce of all sauces, topped with an egg of all things (it works!) sprinkled with parmesean panko breadcrumbs. A few thoughts: 1) i can’t believe i lived this long without trying Meunière, 2) panko and parmesean belong together, on my plate, and 3) you can substitute sweet potato for the egg (i’ve done it and its delicious).
  • Manhattan-Style Fish Chowder – More surprises here like: 1) a soup with fish?! 2) Lemons make the dish yet again, 3) this actually tastes far fresher and is just as comforting as the best clam chowder you could dream up. Note that along with the other commentators, i agree that you should halve the Old Bay seasoning.
  • Shokichi Squash Ragù & Mafalda Pasta – This is a stunning dish if done right. No ingredient is optional. I once proved this by omitting the celery. It is important to have the right pasta, otherwise the noodles won’t catch enough of that gorgeous, creamy butternut squash sauce (im sure Steve Jobs would describe it thusly). And oh thank heavens for rosemary.
  • Roasted Cauliflower Steaks – I cannot think of a recipe whose name more completely belies the supreme beauty and taste it affords than this. This recipe has all the tastes: lemon zest on the lower farro/arugula layer, mild roasted cauliflower in the middle, and a symphony of nutty hazelnuts, sweet grapes, bitter shallots, creamy browned butter and citrusy lemon on top. If you can find fennel pollen, great – otherwise finely chopped fennels seeds suffice. This dish is beautiful: red grapes, green arugula, white cauliflower, brown farro. Like the best blue apron recipes, the ingredients are all familiar on their own, but I doubt in even geological time scales I would have dared combine them in one dish.

I am no blue apron salesman, but I do love the idea of freedom from tyranny – and blue apron frees you from the oppression of bland and boring food. Additionally it provides an exciting escape from the meat-centric, canned-food based recipes of your childhood. Go blueapron!

Uncategorized

Random Opinions of MATLAB

Leave a comment

While working on my Masters at BYU I had a professor that did all his demonstrations in MATLAB. It was as if nobody had told him that they had invented other languages. Or he just really loved MATLAB.

Lately I’ve been doing a project where I’ve had to experience the joy of using MATLAB. As a numerical processing suite it definitely effective: concise, quick, easy. It is also quirky. I thought I’d log a few observations for posterity.

First, the editor has some nice things: autocomplete, lots of suggestions for writing more idiomatic MATLAB. For example:

  • number conversion – i was tempted to use str2num, which does work, but MATLAB suggested str2double, which also works but is mo’ betta
  • string finding – Of course there is a findstr and strfind… you guess which one MATLAB prefers. The MATLAB will correct you. (hint: its strfind)
  • printing – My very first transgression was trying to log statements using disp(sprintf(‘stuff: %s))… For some reason i found sprintf but didn’t yet know that disp(sprintf()) is analogous to fprintf()… but MATLAB wasn’t afraid to correct me

Matlab has a ton of quirks:

  • The editor is SOOO WEIRD. Examples: weird key combos for copy and paste… seriously, Alt+2?? Where are my vim bindings?
  • Performance of deeply nested classes. Deeply meaning depth > 1. I wrote a file reader class. The reader had a data member that allocated a large array. Performance went in the gutter. Then I moved the array to the same level as the file reader, keeping all logic the same, and performance was better. Why oh why would it matter. I don’t know, but it does.
  • Weird behavior when return value not specified. I realized c++ does this. If you don’t specify a return on a non-void function, bad things can happen. I’d expect more from a (presumably) high-level language like MATLAB.
  • paths. I think the notion of having code paths is weird. Why not just have a project with all the paths set up?

Except for the editor, which is appalling, I actually find MATLAB quite usable. I wouldn’t claim it is superior to python, just different.

Uncategorized

How to get an ovpn file onto iOS device via HTTP

Leave a comment

Recently I wanted to get a VPN connection from a phone to an openvpn server. The openvpn app only provides two or three options for getting your ovpn file: 1) use itunes, 2) go to a website in safari that has the .ovpn file, or 3) send via mail.

Option 1 was out since the phone was locked down enough to not permit me to transfer apps. Not sure why – likely just a policy of the phone (i dont own the phone).

Option 3 is less secure so I skipped that.

Option 2 is easy – I only had to make a few mods off what I found via Mr. Google:

  • Make the .ovpn file have a unified format. Some places on the internet suggested you could have configuraiton lines that looked like the following: 
    ca [inline]
cert [inline]
key [inline]

    However this doesn’t actually work. Instead, just delete those lines and stuff the ca, cert, key, and tls keys under the appropriate tags (they have the same name as the config lines)

  • Ensure your web server has the right mime type. The .ovpn mime type is “application/x-openvpn-profile”, and can be added to your httpd.conf like this:


    <IfModule mime_module>
    ...
    AddType application/x-openvpn-profile .ovpn
    ...

I then pointed my browser at the .ovpn file, at which point safari recognizes that you can open it in openvpn. All worked marvellously thereafter.

Bread

Sourdough in Virginia

Leave a comment

Starter
I used the following sourdough recipe from King Arthur Flour. I struggled at first, but mainly because I wasn’t doing regular feedings every 12 hours. Once I did that I had a starter within four days. Also, I found that it was not critical to have unbleached flour. I got a vigorous starter using bleached flour.

Another note: for the first few days the starter really stunk. Then something changed and it started to smell great. I found that after I pulled a cup out for bread, I could just add another cup of flour and half cup of water (i do use filtered, not tap), then a week later I could revive the starter just fine.

Apparently if you are going to be away from your start for a long time you can just dry it out, save the chips. Unproven by me, but they say this method preserves the starter for eons.

IMG_9448

Electronics

My DIY Little Bits

Leave a comment

I whipped up some quick LED modules and my own power module using some proto boards and components I purchased from amazon. I haven’t branched out to more complicated module types, but these ones work well. I think each module cost ten cents or so, rather than the 10 dollars little bits charges.

Note that I put a small button to enable power from the module to the daughter cards. The main reason for this is to avoid wearing down the batteries.

Also, you can see I chose a 5-pin connector. Any module can plug into another module, or directly into a power module (which has two output connectors).

When I do this over again Ill make the following adjustments:

  • Make the outside pins ground, inside pin power. This would completely avoid any problems from plugging modules in upside down
  • Provide an always-on pin – I like the idea of some modules being normally-off. But some modules, like a trinket module, should be always on, since they consume so little power as to not matter

IMG_9592

IMG_9593

Uncategorized

Measuring raspberry pi performance

Leave a comment

I initially had high hopes for the raspberry pi 3 and snapped up three of them. I hoped to see one take the place of my media center, another two for monitoring tasks. Unfortunately I became less than enthused about the pi for a media center as it has no good way to stream of youtube or vidangel. Additionally it couldn’t keep up with streaming some HD videos.

It has now been a few months since my initial disappointment and I decided to try out the new firmware and see if things got better.

The first tests i ran were using iperf. One raspberry pi, rpi1, was left behind at the old firmware. Another pi, rpi2, was upgraded to the latest.

Firmware Kernel
rpi1 1b7da52ec944a9e1691745036966b3b2a48b19e8b (Apr 7 2016) 4.1.21-v7+
rpi2 1e7b8e2c9a7319f7b22869f1334c66e2cfc99f4a (Jun 27 2016) 4.4.14-v7+

Initial iperf test (iperf client running on my macbook, server on each raspberry pi – tests were run independently, on one raspberry pi at a time):

RunRpi1 Rpi2
128 Mbits/sec 39 Mbits/sec
231 Mbits/sec 37 Mbits/sec
330 Mbits/sec 39 Mbits/sec
430 Mbits/sec 38 Mbits/sec

I also tried this with a parallel iperf test (iperf -c -P 10)

RunRpi1 Rpi2
130 Mbits/sec 34 Mbits/sec
229 Mbits/sec 38 Mbits/sec
329 Mbits/sec 35 Mbits/sec
429 Mbits/sec 35 Mbits/sec

I also wanted to test sustained (iperf -c -t <10,120>)

Duration (seconds) Rpi1 Rpi2
10 29 Mbits/sec 39 Mbits/sec
120 29 Mbits/sec 35 Mbits/sec

Even on a udp iperf test (-u on the server, -b 50m on client) rpi1 gets 32 Mbits/sec, rpi2 gets about 39 Mbits/sec. In all cases there is a 5-9 Mbits/sec average higher increase with the new firmware. Pretty significant!

Note that if I run these same tests where my macbook is the client but a virtual machine on my network is the server, I see transfer rates in the 500 Mbit/sec range (the -P 10 test gave 513 Mbits/sec and the UDP test -b 1000m gave 627 Mbit/sec). This at least demonstrates that my infrastructure is more than capable of higher transfer rates. Thank goodness for 802.11ac routers!

It is also worth noting a few differences between the pi and vm tests. One difference is that the virtual machine is on a direct-cabled server, whereas both the raspberry pi and macbook are on wireless. Thus for the VM test, only one traversal of wifi is needed, whereas macbook to pi tests involve two traversals.

Now, if I upgrade rpi1 I would expect the numbers to be equal. Right…?

Strangely, no. Even after the upgrade I see rates in the 30Mbits/sec for rpi1. Digging deeper I found one minor discrepency between the two – i set aside 256 MB for gpu ram on rpi1, but only 128 MB on rpi2. Even after switching rpi1 to 128 MB, I still see the same numbers on both. I noticed I had a wireless keyboard adapter on the slower pi – but removing it didnt affect the transfer rates.

Another difference is that there is a raspberry pi camera on the faster pi – it would be interesting to see if that affects things.

I tried testing the ethernet interfaces and found performance across the pis identical.

At the moment I’m not sure how to account for the roughly 25% higher wifi performance on one pi over the other.

Raspberry Pi

WiFi Performance on Raspberry Pi 3

Leave a comment

I purchased a couple raspberry pi 3s. The major draw is that you no longer need an external wifi adapter – yet the cost is the same as the older pis.

The very first thing I noticed is that ssh traffic was stuttery. This cleared up after performing an apt-get update/upgrade. Everything seemed silky-smooth thereafter.

QNAP

Exporting Paths for QNAP Albums

Leave a comment

As nice as the photo station is inside of the QNAP, sometimes you just want to do your own thing with your photos and videos. But what if they are organized into albums on the QNAP? Couldn’t you just pull down the photos/videos in the album? Or, better, get local paths on your QNAP network mounts? All this, and more, is possible with the following script:

#!/usr/bin/python
import shlex
import subprocess
import os

parms = {}
parms['host'] = "qnaphostname"
parms['login'] = "admin"
parms['albumName'] ="Album1"
localBase = '/Volumes'

cmdBase = """ssh %(login)s@%(host)s "/usr/local/mariadb/bin/mysql -u read --password=read s01 -B  -S /tmp/mysql_mediadb.sock -e 'select cFileName, cFullPath   FROM pictureAlbumTable picAlbum JOIN pictureAlbumMapping picMap ON picMap.iPhotoAlbumId = picAlbum.iPhotoAlbumId JOIN %(mediaTable)s pic ON pic.%(mediaColumn)s = picMap.iMediaId JOIN dirTable ON dirTable.iDirId = pic.iDirId WHERE picAlbum.cAlbumTitle = \\"%(albumName)s\\" and type = %(type)s; ' " """

tables = []
tables.append({'mediaTable':'pictureTable','mediaColumn':'iPictureId','type':'1'} )
tables.append({'mediaTable':'videoTable','mediaColumn':'iVideoId','type':'2'} )

allFilenames = set()
for table in tables:
    parms.update(table)
    cmd = cmdBase % parms
    print cmd
    res = subprocess.check_output(cmd, shell=True)
    # Skip the first row, which has column headers
    hits = res.split("\n")[1:]
    print "Found %s hits" % len(hits)
    for f in hits:
        parts = f.split("\t")
        if len(parts) != 2:
            continue
        filename = os.path.join(localBase,parts[1],parts[0])
        allFilenames.add(filename)
open(parms['albumName'],'w').write("\n".join(allFilenames))

This grabs video and photos using ssh to run the mysql query remotely. QNAP references videos and photos from a very similar set of tables in a mysql database (called s01 on my box). The username and password for this database is read/read. The -B option makes the results tab-delimited, and thus more parseable.

Of course this isn’t super polished or anything; just modify the host and albumName parameters and you should be fine. That is, assuming you’re on OS X. If you are not, modify localBase to point to your mount location for the QNAP shared folder.

Update: I missed this originally, but you also have to pass in the type for each of the two queries. If you don’t do this you invariably get more videos and pictures than are actually in the album. The query is now updated above.

Photo Sharing

2015 Review of Open Source Media Sharing Software

Leave a comment

Open source software rules the internet. The majority of website are supported in some form by open source: be it linux, apache, or some open source library or language.

Surely, then, we could find an open source alternative to social media websites like Facebook, instagram, and Flickr… right?

Here are my requirements:

  1. Server-side import. I have a huge set of media that increases weekly. I need it to be automatically imported – so something server-side must work.
  2. Multi-user. I am hosting storage space for several of my family members. I need a way to have separate logins, preferably with some level of permissions. I don’t want search engines crawling it so “unauthenticated” users should be rejected.
  3. Ability to accept comments. If people have something to say about I photo I want to hear about it.

Here are the options I found:

  • OpenPhoto. So much potential. A Yahoo engineer decided to quit and work on this around 2011 or so. The goal was to provide an open-source alternative to Flickr or Instagram. The project was eventually renamed Trovebox. Even after kickstarter funding, and $1M of additional funding, the project never reached critical mass and was shut down in 2014. It has limped along but really was inactive as of 2015. The details of the venture are given on the OpenPhoto creator’s website.. Huge amount of promise, sad to hear it failed.
  • Piwigo. Looks old school but has impressive lifespan: over 10 years and still in active development. The featureset is correspondingly fairly good and mature. It has support for batch uploading, all kinds of plugins, themes, etc. There are apps for uploading through android and apple. I installed it and think I dislike it least. My only complaint: why does it feel so clunky?! Also, I had to modify my php.ini to include my locale in order for a bunch of errors (which were spewing to the webpage) to go away. I like how you can associate one photo with many albums. It has extensive plugins, including one that lets users add tags to photos. Lots of options for user management, really easy-to-use server-side import.
  • ZenPhoto. Like PiWigo ZenPhoto has been around for a while. It seems to have a lot of the same stuff, and seems slicker, but didn’t have a good comment system (the comment UI took a note from webdesign from some earlier decade. It has like 6 fields to post a comment. I think I could open a home loan with the same detailed form. What happened to single field comment boxes?) and batch import didn’t seem easy or possible. That said its install was vastly superior to Piwigo. Very nice install.
  • Lychee. Looks super slick. Extremely bare bones. New. Unlike zenPhoto and Piwigo, Lychee appears to follow the look and feel of the modern Internet. Work to support commenting and multiple users appears to be underway (and going slowly).
  • ownStagram. The idea here is to replace instagram. It looks like capable software. Not sure how it fares on comments, bulk upload. The UI isn’t that sleek – weird looking ripoffs of instagram images. I didn’t try this one as it didn’t look compelling from the UI standpoint (and has no documented featureset, and only two contributors on github). It does have an android app though!
  • MediaGoblin. I have no idea what is going on here. The concept is cool: federated media sharing. Media includes audio, pictures, movies, etc. I installed it and found that to even get it work I had to deviate from the install instructions and use an older version of flup. Once that was installed I found I was able to upload photos one at a time. No real support for server-side importing (some ideas in the works). I was left scratching my head. I did a quick LOC count – 36k lines of code to do single-photo uploading? I couldn’t get PDF or MP4 uploads to work. A lot of the unit tests weren’t passing. Augh GNU.

Given the state of things, I’d say Lychee is my first pick, were it not for missing multi-user and comment features. For now I’m sticking with Piwigo.