For a project I am currently working on, I need to develop a web and mobile client. In short, it is a data-management system for a government agency where people in the field will report environmental data, while regional and national offices validate the data and use it in management reports.
There are many citizen science projects around the globe where scientists or policy makers try to involve the general public in collecting data. For conservation, citizen science is a powerful tool, allowing policy makers and conservationists to involve the public in helping to make policy, employ a large voluntary workforce and create awareness among the public. Traditionally this has been done by creating dedicated mobile apps, which citizen scientists use to report data. A plethora of apps exist for many different purposes.
The last few months a lot of the media I watch/read/listen to have been up in arms about two topics: Net Neutrality and DRM. Net neutrality means that internet providers should be neutral to the content that people consume through it instead of favoring or limiting some content providers (Facebook, Netflix, etc) over other for various reasons (political, commercial, etc). DRM (Digital Rights Management) on the other hand allows content providers (YouTube, Netflix, etc) to encrypt content on their websites and control who can decrypt it in order to protect their copyright. Advocates of Free and Open Source Software (FOSS), and advocates of an Open Internet have been crying out against the developments with the US FCC proposing to eliminate net neutrality and with the W3C (World Wide Web Consortium) about to adopt a DRM standard called EME (Encrypted Media Extension).
The last few weeks I have been busy on a solar monitoring project. I have a minimal viable product working based on an RS232 enabled inverter I have here. The first client I wanted to deploy it with has a similar inverter from the same company, just with a usb port instead of RS232. I was expecting it to be a serial over USB port that would just work as-is with a usb to serial driver, especially because it advertises itself as a Cypress USB to Serial device when you attach it to your computer. Unfortunately this wasn’t true, requiring me to reverse-engineer the USB communication for it. In the end this was easier than I thought it would be and it was a whole lot of fun. This post just shares the quest I went through.
So I love AngularJS (or Angular as you’re supposed to call it now) for creating the interfaces of web-applications. I have really started to decouple the interfaces I make from the backend, by using Angular for the front-end and flask to create a RESTful backend with which Angular communicates. It’s awesome. Right now I am making a web-interface that draws an image and dynamically changes numbers in that image depending on data it fetches from the backend. It is used in this case to show an image of a Solar Power system and dynamically change numbers in the image, showing the current state (battery capacity, panel output, power consumption, etc) of their system. I am using Angular for this and the HTML5 Canvas object.
Recently I needed to reverse-engineer the serial communication between a solar inverter and a computer. Although hardware manufactures may often be good at making the hardware they sell, their accompanying software often leaves a lot to be desired. To develop my own software, I needed to interface with a specific inverter, both to intercept the serial communication at first, but later to write my own software. In order to be able to write unittests for the serial port, and to test my software without actually being connected to an inverter, I wanted to have a dummy serial port on my computer that listens for any connections to the port and responds whatever I tell it to.
I am working on various electronics projects where I want to have a sensor node that collects data from a sensor and either logs it to a micro-SD-card directly or sens it wirelessly to another machine that uses/logs/etc the data. I opted for a set of NRF24L01+ boards for the wireless communication between the nodes and data loggers. So to test it out I want to communicate between my Beagle Bone Black and an Arduino. The NRF24 library by TMRh20 supplies a unified API for the NRF24L01 that should work on Arduino, linux (like Beagle Bone Black and Raspberry Pi) and other platforms. It also includes some examples that are identical across platforms so you can test between platforms with the same code. Really cool. I had trouble getting it to work on my Beagle Bone Black. This post hopefully will help other getting it to work on a Beagle Bone Black running Debian Jessy.
It’s been a while since I did a proper post about something techy. But I did spend quite a bit of time with Knime, and I love it. So I think it warrants a post. Knime is a complete data mining, manipulation and analysis toolkit. It’s completely open-source and really flexible. I have experience with other ETL and reporting tools, like Talend Open Studio, JasperReports and Pentaho Kettle but for me (in my use-case) Knime rises above them. Talking about my use case, what is it?
The Tambopata-Madidi area on the foothills of the Andes is one if the last areas of Amazonian forest that is relatively untouched by man. It contains 42000km2 of national parks, ranging from Tambopata in Peru to Madidi and Pilón Lajas in Bolivia. It is home to two important tributaries of the Amazon river and an incredible biodiversity with both montane and lowland tropical forest, harbouring over 10% of the world’s bird species, and possibly the most biodiverse place in the world. Around 15000-30000 people divided among three different indigenous peoples call it their home, with an interesting cultural diversity, bridging the Highland people for which Peru and Bolivia are famous to the lowland Amazonian tribes.
For my aquaponics project I need to create a timed switch that turns my pump on and off. I am doing this with a simple 555 timer in a-stable mode, a capacitor, a relais and two pot-meters. The timer in a-stable mode switches a pin between high and low state, for a fixed amount of time. Both the high and the low times can be adjusted with the potentiometers.
Since a few weeks my laptop battery is dead. I am still trying to work with Asus to get a new one as it is still under warranty (but sending it in for repair will apparently take 2 months while I already know that a new battery will solve the problem). So I am working of the net power only. The power sockets in the Philippines aren’t great though, and Asus decided to integrate the plug into the transformer/regulator unit for my laptop charger, creating a big heavy plug that is really too heavy for vertical wall sockets. So as you can guess from the title of this blog post, yesterday I was working on my laptop and decided to install upgrades on my Arch Linux box. While I was doing this though, I accidentally touched the power cord of my laptop, which then fell out of the wall socket. This isn’t the first time this happened, but unfortunately my computer was in the middle of a kernel upgrade…. And of course the kernel modules weren’t fully installed, and the initramfs wasn’t fully updated either.
This post is probably a no-brainer for quite a lot of people. And it should have been for me too. I don’t know if my brain was just not working from the dry and hot el niño weather, but it took me quite a while to figure this out. Since chances are there are others with the same problem, I thought it’d be worth to write it up.
The USGS isn’t just a scientific organisation doing a lot of satellite imagery and remote sensing work. They show themselves to be artists as well. These images are absolutely gorgeous. Earth itself is gorgeous and there are many ways of looking at it. The way the USGS Landsat team are looking at it is definitely way out there. The way I use the same images to try to look underwater is a bit more mundane unfortunately. These are just some of the images. You can see more on the USGS website.
Little by little I am working at my current job on an Aquaponics system together with the help of our volunteers. As a final step to the first implementation I need to regulate the flow of water from our fish tank to the grow bed. I decided to do this with a timer that switches the pump on and off. In order to create the electronics for this timer, I need to know accurately how much water is flowing from one compartment into the next. Of course I could just have done the maths for this on the back of a napkin, but I though writing an application that does the simulation would be much more fun.
I love Gnome Music Player Client (gmpc) as an audio player, with MPD (Music Player Daemon) as a backend. Both because of the remote control possibilities but GMPC is also just a great client. But when installing MPD, combined with PulseAudio on my Arch Linux laptop, I ran into trouble. MPD runs as a different user, and as that user wasn’t able to connect to PulseAudio that runs under my regular desktop user.
Recently I have been analyzing a large amount of data with R. A great tool to do this is Rstudio. It is an IDE for R that makes it easy to write your R code, explore the data and show the graphs. But if you want to communicate your results with others, sitting behind an IDE isn’t the best way. Fortunately Rstudio integrates with Knitr, RMarkdown and Pandoc. With those tools you can create PDF, HTML or word files from your R code.
I am running Arch Linux for a while now and am quite happy with it. Even though it is not the easiest n00b distro, in the end the documentation is so good that if you’re new to Linux, you can probably find out how to do most stuff if you’re inquisitive. One thing I had trouble with though, is getting a custom Live CD that could boot from my USB drive, has a persistence layer so changes are saved when booting, and booted both from UEFI only systems (my laptop) and preferably also BIOS only systems. It took me a while to figure this out, but I got it in the end. So I’ll document it for posterity.
For this post I want to share some of my favorite web-application development tools. When I started thinking about this though, I realized there were so many, and which I use depends so much on what it is that I am trying to do, that I decided to take maybe a somewhat strange approach to this. I am going to approach it as a comparison on two sets of tools: python web development tools vs php CMS-es.
Over the years I have wondered why researchers still rely on spreadsheet applications to record their research data. Although most people know how to use them, spreadsheets have very serious drawbacks, especially if multiple people contribute data to the same dataset/project. With spreadsheets it is usually not possible to have multiple people enter data concurrently. It is also hard to check the sanity of entered data unless you start working with complex formulas and macros, which quickly become very complex, error prone and hard to maintain. And lastly, transforming data in a spreadsheet where you combine data from different tabs/tables, transpose data, filter/sort data, group/split data, etc becomes hard and error prone. For these (and more) reasons, (relational) databases with a decent data entry application in front of it are much better suited for research data.
Over the years I have (happily) setup and used many different ways of hosting personal and company email and websites from Zimbra, commercially hosted virtual servers, in-house servers, cheap webhosting companies, servers at home and blogs and websites in blogger, Drupal, Wordpress, etc. All of them have their merits and drawbacks. But since I moved to the Philippines and started working for Marine Conservation Philippines I was looking for something cheap (free?) and low-maintenance to host the email from my old domain name, and host a weblog for personal ramblings. This is what I ended up with.