Programming overview

Here you can follow my progress in programming. This article also provides an overview of the other programming-related articles by DerAndere.

Bio7 / Eclipse: An IDE for programming

For the purpose programming, one can use powerful editors such as Emacs (open source) or integrated development environments (IDE) that can be extended by plugins for various use cases that add graphical user interfaces. I am interested in a) programming microcontrollers in C and C++, b) creating python scrips and using the programming language R for data processing and image manipulation and c) maybe learning a bit of Java. One software framework which supports all this is Bio7, an open source Eclipse rich client platform (RCP) which is based on the popular IDE Eclipse and integrates the Eclipse plug-in GEF that adds a flow chart editor as well as. This makes it possible to handle DOT files. After installing the Eclipse marketplace client plug-in, the Eclipse CDT (C/C++ development tooling) plug-in as well as the plug-in Sloeber (via the Eclipse marketplace), Bio7 is set up for efficient physical programming of developmentboards which are supported by the Arduino IDE (e.g. development boards with a Microchip AVR microcontroller such as Arduino UNO compatible boards, development boards with a SAM microcontroller such as those compatible to the Arduino Due, but also ARM Cortex-based development boards such as the Teensy USB development boards by Paul J. Stoffregen (http://www.PJRC.com). Sloeber is an extension for the Eclipse CDT plugin. A video showcasing basic features of the Eclipse IDE such as error detection, code refactoring, version control, auto code complete and search/browse is available online. Sloeber comes bundled with MinGW which provides the GNU compiler collection (GCC), as well as the AVR-toolchain including AVR-GCC and AVRdude and the GNU Debugger (GDB). It adds a toolbar for straight forward physical programming: Simply click on a button to upload sketches onto microcontrollers. Make sure that the Arduino IDE and Java Runtime Environment are installed if you do not use the Sloeber product bundle which includes those and Eclipse + the C/C++ unit testing support. If you did not install the Sloeber product bundle but the plug-in only, make sure to left-click Help -> Install new software. Select “All available sites” from the dropdown menu, type “C/C++ unit testing support” (without the “”) and left-click “go”. Install one of the results called “C/C++ unit testing support”.

You can add further plugins to Eclipse that enable programming in Java (Java development tools, JDT, in Bio7, simply select the menu Preferences -> Preferences -> Preferences Bio7 -> Preferences Custom and activate the option “Enable JDT Gui elements” or the development of graphical user interfaces (GUI) using Qt (Qt integration for Eclipse or the plug-in The Complete Eclipse C/C++ IDE) and plug-ins that allow to embed C++ code within code that is written in R (Rccp + RInside, http://blog.fellstat.com/?p=170). On top of that one can install the eclox plug-in bundled with doxygen for code documentation. The plugin Egit allows for git repositories be cloned in order to create a local repository fork and perform version control. The git account information can be added via Preferences -> Team -> Git -> Configuration -> Add. Key: user.name, value: GitHub user name Apply. Again add an entry, key: user.email, value: Email of Github account. The Eclipse plugins Mylyn Gitlab connector can be installed additionally. Take your time to follow the Egit user guide and its tutorials. The plug-in DBeaver serves as client software with a graphical user interface which provides functionality to access and manage databases such as MySQL or PostgreSQL from within Bio7 / Eclipse. The plug-in PyDev adds tools for writing Python scripts. To enable the functionalities of the R perspective of Bio7, Left-click proferences -> Preferences -> Bio7 Preferences -> Preferences R. In the section “Packages” Left-cllick the dropdown menu “Select Server” and choose a Server that is close to where you are (for Germany, select Germany: Goettingen). Then, left-click “Apply and Close”. Finally, start Rserve: Do not use the Bio7 main menu R -> Start Rserve, instead use the hotkey (<Ctrl> + <r>) or the tool “Start Rserve” in the Bio7 toolbar or from that tools dropdown list the entry “Start/Stop Rserve”. If you do not need the tight integration of imageJ within Bio7, you can install the Eclipse plug-in StatET which serves as an alternative IDE for development in the programming language R with a graphical user interface. The plug-in DocBook editing and Processing Features for Eclipse (DEP4E) adds further documentation features for creation of DocBook XML-based manuals and documentation in the form of help files and the plug-in Vex adds visual XML editing capabilities.

Programming in C++ using the Bio7 / Eclipse IDE with the Eclipse CDT plugin

To create a new project, select new -> C++ project from the Eclipse menu. Enter the name of the Project (Projectname) as the filename. After selecting OK, a new project with the name Projectname is generated and in the folder where the eclipse workspace is saved, a Project folder that is named like the Project is created. This project folder contains a new program main file with the name Projectname.cpp. (.cpp is the file extension for source files written in C++). In the Project explorer within Eclipse, you will see the project folder with its contents.

Before learning how to tell the machine what to do, it is important to know how to make programs as well readable to humans. This allows for code maintainance and resolving of errors (bug fixing). It is advisable to learn about the principles of doxygen code documentation (e.g. using the eclox plug-in for Eclipse).

I use the following for special documentation block for a on line brief and a longer detailed documentation before modules, classes, functions etc.:

/**  
 * Brief description.  
 * long description.  
 * @fn function description  
 * @param parameter description  
 */

In addition, non-intuitive lines of code can be explained by //< in-line comments

in the same line after the code (at the right hand side). Also, indentation enhances readability.

Functions and variables are given short but descriptive names in camelCase with the first letter being small. Variables that serve as loop counters are i (j for the next nested loop).

Variables that serve as an array index can also be short (e.g. single letter).

C++ is an object-oriented programming language. Objects are variables, containers such as arrays, vectors or lists and functions. Each object has to be declared by specifying its type and its identifier (an alphanumeric name without a space). The function int main() {} serves as the entry point for each C++ program. (Even the Arduino IDE adds the main() {} function during the conversion of the .ino file). The code inside the brackets of the function main() {} will be executed. A simple program would be

/**
 * this is a simple program witten in C++
 */

/** the following code can be outsourced into a header file called e.g. Projectname.h, if the main file 
 * Projectname.cpp contains the code #include "Projectname.h" at the beginning.
 */
namespace Projectnameutils {
    int varA; //< declaration and definition of an object with the identifier
                     //varA which
                     //is of the type integer
    int varB; //< declaration and definition of an object with the identifier varB
              //which is of the type integer
    int valC; //< declaration of an object with the identifier valC which is of type
              //integer
    int func(int /* var1 */, int /*var2 */); //< declaration of a function prototype
              // for a function that returns the sum of var1 and var2 as integer
}


/** the following code has to remain in a source file which contains the
 * implementation. Names of source files end in .cpp
 */
Projectnameutils::varA = 2; //< initialization of the object varA by assignment
                            //of the value 2 outside namespace Projectnameutils
Projectnameutils::varB = 3; //< initialization of the object varB by assignment of
                            //the value 3 outside namespace Projectnameutils

int Projectnameutils::func(const int var1, const int var2) { // <definition of the
            //function Projectnameutils::func() outside namespace Projectnameutils
    int sum = var1 + var2; //< definition and initialization of a local variable of
                           //type int with value calculated from var1 and var2
    return sum
}

Projectnameutils::valC = Projectnameutils::func(Projectnameutils::varA,
        Projectnameutils::varB);. //< Initialization of
                                  //the object valC as a function with two previously
                                  //defined arguments: valA, valB.

int main() { //< entrypoint of the program that is executed in hosted env. (OS)
    int valD; //< declaration of the local variable valD of type integer
    int valD = func(Projectnameutils::varA, Projectnameutils::varF); //< definition
                //of the object valD as a function func of valA and valC that were
                //previously declared.
} //< main() ends, end of the program

The above program defines three global variables in a namespace. It assigns the values 2 and 3 to varA and varB. To varC it assignes the sum of the firs two variables to which the values 2 and 3 were assigned. At runtime, the local variable varD is declared and then defined as the sum of varC (value: 5) and varA. The result is assigned to varD. The user of the above program cannot see the result, but he can write further lines of code that can manipulate the object valG and its assigned value, which is 7 at the end of the above code. The namespace reduces the risk of unwanted effects if the global variables are used externally. Because all values are constant and not used outside of the translation unit (this file), all variables could be changed to const or even constexpr objects. Those have static (internal) scope, which is limited to the current translation unit.A constexpr is evaluated at compile time.

For more diverse functionalities, header files from standard libraries can be included at the beginning of all files that use their functions. Therefore, add the following code at the top of the project main file Projectname.cpp as needed:

#include <cmath> //< provides mathematical operators
#include <vector> //< provides associative (resizeable) vectors. See <array>
#include <iostream> //< provides standard input/output stream objects cin, cout
                    //< (object oriented, safe in comparison to <cstdio>. Use it with
                    //< boost::format and std::ios::sync_with_stdio(false);
#include <cstring> //< provides functions to manipulate C strings and arrays
#include <cstdio>  //< provides printf(), file access via fopen(), fclose()
                   //< (use iostream, if there is no problem with speed)
#include <ctime> //< provides clock(), time(), difftime()
#include <boost/format.hpp> //< provides formatted output with <iostream>

These lines of code include C++ headers from the C++ standard library. A huge variety of C++ libraries that provide fast access to relevant functions can be found online. The most important ones are usually included in the editor / IDE. The most popular one is the C++ standard library that also contains headers from the C standard library, which in turn integrates parts of the well known standard template library (STL) that was developed at the Hewlett-Packard research labs and released into the public domain in 1994. It provides associative containers such as vectors, lists and arrays with variable size. It’s features are declared in the namespace std::.

Refer to the documentation for the chosen library for usage and syntax. For examples, use vectors with the following syntax:

#include <vector> // provides std::vector container of the C++ Standard Template  
                  // Library

std::vector<T> (n) {element0, element1, element2, …, element(n-1)} vectornameA

where T is the type of the vector elements and n is a size_t (unsigned integer) defining the size of the vector (the total number of elements in the vector). vectornameA ist the assigned name. std:: is the namespace that makes sure, the function vector is called from the header <vector> of the C++ standard library.

Below is another example:

#include <vector>
#include <iostream> // provides cout << for printing via the standard output stream

std::vector<T>{4,5,6,7} vectornameB
vectornameB.size() //< gives the size of vector vectornameB as a size_t (unsigned
                   //< integer). This can be saved in a variable:
size_t varA = vectornameB.size()

void printVectorElements(auto &vector); //automatic type deduction,pass by reference

void printVectorElements(auto &vector) {
  for (const auto &i: vector)) { // range-based for loop (since C++11)
    auto currentElement = vector[i]
      cout << currentElement << endl;
    }
  }

int main() {
  printVectorElements(vectorA);
}

Other standard libraries can be included using #include <Headerfilename.h> where Headerfilename has to be substituted by the name of the library of interest. In general, variables and functions that are used in several files and different projects should be moved into a library. Custom libraries can be created and used as described on the page Programming microcontrollers of this blog.

Programming in Python 3.7 using the Bio7 / Eclipse IDE with the PyDev plugin

A Python project usually consists of the project folder containing the project’s LICENSE file and a source folder (src). The src folder contains a Python package. The package is a folder (in the Bio7/Eclipse IDE with Pydev, this folder should have “Pydev nature”) containing an __init__.py file that only contains minimal initialization code (metadata about the package) and so called modules. Modules can be either Python packages themselves or Python scripts (text files with file extension .py). If a Python program is meant to stay running until it is closed, all code that needs to be it executed in the main loop has to be inside a function called e.g. main(), in combination with the following snippet:

if (__name__ == __main__):
    main()

Usually that code snippet is inside a Python script with the project name or “main.py”.

Use object oriented programming to avoid global variables.
If you still need global variables, put them into a namespace
Use from foo import bar, fbar or import foo instead of from foo import *`
Use composition instead of inheritance, especially if the relationship between classes is possessive, i.e. you can create a composite class that has one or more objects that are instances of child classes (instantiated with default or non-default behaviour)
Use inheritance only if the child class really is a subtype of the parent class that merely extends it so you can strictly follow the Liskov substitution principle. The principle states that “in a computer program, if S is a subtype of T, then objects of type T may be replaced with objects of type S without altering any of the desired properties of the program”. If the default behaviour of the parent class is different from the default behaviour of the child class, you should not have used inheritance but composition. More information can be found online.
Always explicitly mention the names of keyword arguments.
Use **kwargs whereever possible. Keyword arguments are much more self-documentory than positional arguments (*args).
Use implicit Python 3 style of super() (and using **kwargs) for inheritance with indirect reference!
Use f-strings if no punctuation inside strings is needed.
Use the Python library “subprocess” with the subprocess.run() API for executing commands in the command in the command interpreter of the operating system (e.g. the Linux terminal). Under Microsoft Windows, the Python package “subprocess” sends commands to whatever the value is for the %COMSPEC% environment variable in the running process, falling back to cmd.exe if there’s no value there. That value can be changed to powershell.exe if preferred). With the subprocess library, provide the command (parameter cmd) as a list of strings and do NOT set shell=True for security reasons.

If you develop graphical user interfaces, use either pyQt or wxPython. If you choose wxPython, be aware that a lot of online resources use bad style. Here are guidelines for using wxPython:

Create buttons procedurally using lambda or the partial functionality of the functool library instead of manually creating them one-by-one. An example can be found here (in combination with this function call).
wx.grid can be used in combination with a wx.button to implement multi-selection. And example can be found here.
The wx.html.HtmlWindow() might be advantageous compared to wx.TextCtrl(style=(wx.TE\_MULTILINE | wx.TE\_RICH | wx.TE\_AUTO\_URL | x.TE\_NOHIDESEL | wx.TE\_READONLY)) or wx.richtext.RichTextCtrl(style=(wx.richtext.RE\_MULTILINE | wx.richtext.RE\_READONLY)) if you want to handle preformatted HTML
Always explicitly mention the names of keyword arguments!!! This is key (pun intended) for writing readable code, although even the style-guide of the wxPython wiki does not take care in this regard. Otherwise, follow that style guide. To be more precise:
If applicable use wx.BoxSizer as opposed to other sizers or hard-coding positions
Use SetInitialSize(size=wx.Size(sizeX, sizeY=-1)) for defining window sizes in cases where the layout (here: sizeX) cannot be created solely by means of using sizers.
Create discrete custom classes instead of nested panels defined in place

Pages on further topics can be found in the navigation bar at the left side of this blog:

History and theory of computers

On the page History and theory of computers, I give an overview over the historical development of early computers and I provide a theoretical background starting with literature from the research fields of quantum chemistry and surface physics that provide insight into the fundamental processes relevant to semiconductors and doping. Further, you will get information on how doped semiconductors form the basis of transistors which are in turn the building blocks of logic gates that constitute arithmetic logic units (ALU) at the core of central processing units (CPU).

Fig 1

Version control and collaboration

On the page Version control and collaboration, I give tips on how to use the Eclipse plug-in Egit as a frontent for the version control tool git.

Programming microcontrollers

[Werbung ohne Auftrag / unpaid advertisement]

On the page Programming microcontrollers, I write about the significance of microcontrollers, about alternatives and about how to get started tinkering with microcontroller devices. I will focus on setting up the integrated development environment Bio7 / Eclipse and the Eclipse plug-ins Sloeber and the AVR Plugin for programming and debugging / unit testing Arduino-compatible microcontrollers in C/C++. I will present my first two projects that involve microcontrollers: The All Yarns are Beautiful (AYAB) shield for controlling a knitting machine from a personal computer and a do-it-yourself MIDI-controller. The latter serves as a test-bed for different programming paradigms including modular- and object-oriented programming.

Fig 2

Fig. 2: Schematic circuit diagram for the MIDI controller by DerAndere. Created using the free version of EAGLE by Autodesk Inc with the Adafruit EAGLE library. Arduino UNO R3 can be replaced with any compatible development board. Copyright 2018 DerAndere. This image is licensed under the terms of the Creative Commons Attribution 4.0 International license (CC BY 4.0).

PipetBot-A8

[Werbung ohne Auftrag / unpaid advertisement]

On the page PipetBot-A8, I present my progress in converting a Anet-A8 3D-printer into a cartesian robot that can handle liquids. The goal is to create a pipetting robot for usage in a chemistry lab and to learn how to program a cartesian robot using G-code. Furthermore, I explain how to expand the Anet-A8 based liquid handling robot with additional periphery and present the program “Graphical G-code Generator” (GGCGen) that I wrote in the Python programming language in order to provide a graphical user interface (GUI) for easy programming of cartesian robots.

Fig 3

Data analysis

The page Data analysis provides an overview over methods for Image analysis and data mining using ImageJ, R, and Python from within Bio7 / Eclipse. In a second section I will introduce the usage of databases for storage and management of data. I will focus on the setup of the Eclipse plug-ins PyDev, DBeaver and the relational database management system (RDBMS) MySQL and explain how to create a relational database using MySQL and the Eclipse plug-in DBeaver as well as how to retrieve data from that database using SQL queries. Additionally, I briefly show how to use the R package dplyr (with the backends dbplyr, RmySQL and DBI) and the Python packages SQLAlchemy (and the backend mysqlclient) for retrieving / processing data that is stored in a MySQL database.

DERANDERE