Cargo build freezing…

Occasionally cargo build freezes near the last step (26/27 for me) and i ctrl-c. Then running cargo build again I get this error:

Blocking waiting for file lock on build directory”

Searching the internets people claim this can be fixable by clearing out the package-cache (“rm -rf ~/.cargo/.package-cache”), however this doesn’t work for me. What DOES work is cargo clean then rebuild. This takes WAY too long. A better way is to just pkill the running process (ps aux | grep cargo, or pkill cargo).

rust cxx crate: linking to external c++ shared lib

I have a custom c++ shared library. I also have a rust application that needs to use that library. I had no problem making a build.rs using bindgen, however I wanted a little better c++ support so I turned to the cxx crate. All went well until link time as I was getting “unresolved symbols”… but the linker output showed the library WAS being linked in! What gives?!

First, here was my source BEFORE the fix:

    // linking to c++ "libcamlite"
    println!("cargo:rustc-link-lib=dylib=camlite");
    // .... code removed for brevity
    cxx_build::bridge("src/main.rs")
        .file("cxx/wrap.cxx")
        .std("c++20")
        .debug(true)
        .compile("libcamlite-rs");

The solution? Behold!


    cxx_build::bridge("src/main.rs")
        .file("cxx/wrap.cxx")
        .std("c++20")
        .debug(true)
        .compile("libcamlite-rs");
    // .... code removed for brevity
    // linking to c++ "libcamlite"
    println!("cargo:rustc-link-lib=dylib=camlite");

YES… I simply moved the cxx::build command above the link directive…! Seems like a bug, eh? And I don’t have time to care much more about it – but hope this helps someone.

Tesla Model Y After 4 Years

About 4 years ago I bought an early edition of the Tesla Model Y. I immediately loved it. I still think it is the best car available.

When I purchased the car I also paid for Full Self Driving (FSD). It was understood that FDF wasn’t actually ready – but that it would mature over the coming months… or years…

Steady progress in FSD was made – but it seemed to stall for about a year and a half. Every new release would be proclaimed as “mind blowing” – but was only a small step change. This all got old. By summer 2024 I was pretty well resigned that there was no reasonable timeframe in which one would expect real progress.

This all seems to have changed in FSD 12.5.4 – for the first time, it seems smooth, doesn’t have the nag that just annoys. Doesn’t beep at me. It just drives. It’s really nice.

I hope the Elon haters just try to let the car drive you. Congrats to the Tesla team for sticking to it!

Apple Watch Solo

Apple makes some good products… I’m not so sure the Apple watch can be counted as one of them. Reasons: the battery life isn’t stellar, and, most importantly, it is an utter pain to set one up for family member.

The setup problem is because apple requires an iphone to set up the apple watch and, to travel the royal path, said iphone should be on a plan that supports the apple watch. Since I am cheap, I am not on one of those plans.

After much pain and wasted time I found the one true way to connect the watch to cellular while keeping a cheap plan on your phone. The process is superficially easy, but unnecessarily painful. Roughly it goes like this:

  1. Call verizon. Yes. Call. In the year 2023 – you MUST call them.
    • You must connect to their “inside sales” departemnt
  2. Tell the sales associate you want to connect an apple watch to cellular in standalone mode
  3. Given them the watch IMEI
  4. Ask them to activate the watch. Then activate the watch using your iphone. Do this with them on the phone
    • If they don’t activate now, you will try endlessly to get it to work in vain. You will call them the next day and have to pick right up where you left off.

Note to self: Even rocky 9.2 uses bridge-utils

Sometimes I decrypt LUKS volumes via dracut/dropbear (ssh). To enable this I follow the instructions from the dracut-ssh project. However since I usually use my linux boxes as hypervisors, and vms need a bridge, I end up with bridged networking.

Now for the problem: if your dracut networking config does not align with your normal post-dracut networking, you end up with a weird blending of the two. So to avoid this I just keep dracut the same, building by bond/bridges in dracut to match the normal configuration.

Here’s a sample portion of my /etc/default/grub:

rd.neednet=1 ip=10.10.2.2::10.10.0.1:255.255.240.0:server-2:bridge0:none:8.8.8.8 bridge=bridge0:enp47s0

This all works well, so long as you are sure to install bridge-utils – otherwise dracut silently fails.

yum install -y bridge-utils

Be sure to rebuild your dracut

dracut --force --regenerate-all

Viola! Fin.

Disneyworld sucks

It’s not magical – no matter how many times they say it is. It is as magical as having a pickpocket empty your wallet everyday and giving you back only lost days of your life and a sunburn.

I hope someone starts a new theme park that is actually based on some new ideas and real entertainment, not hackneyed recycled flufff from yesteryear. Walt would be so ashamed.

Review of Starlink Internet

I have been using starlink for several months and have been thoroughly impressed! I switched to Starlink even though I live in an area with multiple high-speed internet options: Verizon FIOS, Comcast xfinity, etc… The “big boys” of internet. They all tout multi-hundred Mbit/sec internet downlink; in the case of fios symmetric uplink. Starlink, in the other hand, barely breaks 100mbit/sec down. So why bother with it?

Normally you don’t care about your ISP – you just use your internet and it’s great… but what about when it isn’t so great?

Enter exhibit A – fios performance against starlink. Fios basically went to pot around january. No manner of rebooting of routers would resolve it. Starlink didnt tank – just verizon.

Now enter Verizon’s consistently crappy service: i call verizon and get bounced between 4 or 5 different “agents.” Nobody could help – but they acknowledged the problem. They assured me they valued me as a customer. Then they started blaming my router… but in my case, they didn’t realize i have redundant routers running pfsense. I’m not a black belt network engineer but I am fairly capable of handling my home internet router. I tried in vain to explain to them that it wasn’t my router. This proved a waste of time so I asked them to cancel. Even getting them to cancel took forever! And to top it off they informed me that since I was cancelling at the beginning of a billing cycle i would still have to pay the full amount – no proration – “per the contract.” Well, so much for valuing me as a customer – they don’t care and never did, even up to the very bitter end.

Now enter starlink. Normally one has no bargaining chip if they have a single internet uplink. I have pfsense load balance my internet across starlink and fios. This meant I could cancel Verizon without any interruption of service. So satisfying.

Granted Starlink is NOT as fast as FIOS. Not even close. But the fact that “Starlink != Verizon” is good enough for me. I’m even willing to pay more, obviously; starlink is now $120/mo for me, and verizon was only $80/mo. That’s what sheer disdain does.

Oh and user experience with starlink is way better. Everything is managed out of the starlink app. Want to cancel? Upgrade? Just hit the button, for goodness sakes! And never are you required to interact with anyone to set it up, change, or shut it down. Take a hint Verizon – nobody wants your service reps.

Also starlink is improving: Around march the already-low latency dropped another 10ms. See in the image below around March.

Now for the bad… obviously uplink is weak – single digit Mbit/sec. Also during torrential rains I lose internet for a minute or two at a time.

Elon: I’m hoping starlink can add a cheaper tier – perhaps $60/mo? I’d even be good with $60/mo for 60 Mbit/sec downlink? But please don’t drop uplink speed. Its already too low.

Probably time to dump Redhat

Late in 2020 Redhat made its first attempt to kill CentOS. Then came the heroic rescue of Alma and Rocky distros. Now just last month, June 2023, Redhat is attempting to kill these “downstream distros.” They – wrongly, I might add – assert “recently, we have determined that there isn’t value in having a downstream rebuilder.”

We could argue with redhat or just move on. There are other distros. But the tragedy here is that redhat is missing the entire point of open source. Everyone contributing anything that worked on CentOS was bolstering Redhat’s offering. It is arrogant for anyone, Redhat included, to view the community as a bunch of freeloaders. Opensource isn’t narrowly defined as sharing of code – it is a community of sharing. In that light Redhat is saying they are done sharing. So they are done with open source. So I am done with them. Time to move on.

The only question now is which distro will be used next. In the past I’ve avoided other distros because there was simply no compelling reason to switch. There is now a reason to switch. Perhaps Ubuntu or Debian? Arch?

Goodbye Redhat – have a good time sliding further into irrelevance.

GPU + TPU parallelism

I use my RTX for heavier ML workflows, like training and high-quality inference (say, centernet). Additionally my security system has edge-based (coral TPU) cameras (one TPU per camera) that perform their own inference workloads. Thus all my compute resources are busy burning any solar power I get, and then some.

This poses a problem as it means i have no spare compute to handle any remaining workloads. For example, during regression testing of new models I need to re-run inference against old data. Neither the camera TPUs nor the RTX is available. Thankfully, one can just add extra TPUs to the main workhorse server to gain additional inference capacity. Each of these TPUs is capable of 90ish inferences a second of a mobilenetv2-based net. All this on the same server performing inference and training using the RTX without impacting performance.

Each TPU operates independently. Currently I have two TPUs in addition to my RTX. CPU utilization sits around 25% while the TPU and GPU resources are pegged. A busy box is a happy box.

For reference, a good docker container for running coral: https://github.com/pklinker/coral-container.git – the scripts provided for objdet are good – just remember you can pass “@:0” or “@:1” to reference your 1st, 2nd, etcth TPU.

Putin is a Sad Bad Banana

How sad that a grown man – a single sorrowful, pitiful human being – could dash the hope and optimism of the world by plunging it into a stupid war. We all make dumb mistakes – but very few of have the distinction to make a mistake that causes the following:

  • Young children, who should be in school and playing on playgrounds, are being killed, scared to death, and left fatherless. Instead their schools are bombed and they are living in a hell zone.
  • Mothers who should be receiving the best of care to bring up their children are dying – some in maternity hospitals. This because in war there is no real safety and everyone is potential “collateral damage.”
  • Young people, in Ukraine and Russia, who should be stretching their minds and talents to the benefit of society, are wasting their lives in a pointless war of one man’s doing
  • The world, instead of solving great problems, like traveling to infinity and beyond, and helping the poor and the sick, have had their hopes reset with the realization that a modern-day madman can bring the storms of war to everyone’s doorstep.

These are the things that fall squarely on Mr. Putin. As such Putin earns, with ignominy, the “sad banana” award – he’s just a gross, rotten banana that nobody wants. How very sad.