samedi 7 avril 2018

Download files with Python, Selenium and Chrome headless

I have recurring tasks these days that consist to automatically download files on the Internet. Usually, requests, beautifulsoup and some tricks do the job effectively. Sometimes though, I have to play it hard and ask Selenium and Chromium* headless to do the heavy lifting. Alas, asking Chromium to automatically download files is not clear.

I found the solution in Chrome tracker, and you read it bellow written in Python:

from selenium import webdriver

# Let's create some option to make Chromium go headless
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('disable-gpu')

# Launch the browser 
browser = webdriver.Chrome(chrome_options=options)
download_dir = tempfile.TemporaryDirectory().name
os.mkdir(download_dir)

# Send a command to tell chrome to download files in download_dir without
# asking.
browser.command_executor._commands["send_command"] = (
    "POST",
    '/session/$sessionId/chromium/send_command'
)
params = {
    'cmd': 'Page.setDownloadBehavior',
    'params': {
        'behavior': 'allow',
        'downloadPath': download_dir
    }
}
browser.execute("send_command", params)

There you go, happy scraping!

* Of course, it works with regular Chrome too!

dimanche 14 janvier 2018

Some talks

I'm watching tech talks less often these days. Most of the time, the talks are too long (45+ minutes). Nowadays, I don't even try to watch a talk about new piece of software or framework: if I want to spend my energy on these topics, I'd better play with an online tutorial.

Here are two talks videos I watched and liked lately. First one is Hammock Driven Development by Rich Hickey. He is the creator of Clojure programming language. In this talk, he explores the fact that we have 2 modes of thinking, the first in conscious and the second is unconscious and work by itself. It's about these "Eureka" moment that we experience as developers when we are away from the keyboard. You'll see why the hammock is a smart choice to enforce unconscious thinking!

The second talk is by Greg Young, who is known for his work on CQRS and Event Sourcing architecture patterns. He demands us to stop over engineering. It's OK not to automate all the stuff by programming. It's about the struggle the desire to show the power of our programmer brains by building over complicated frameworks.

Good watching!

PS: if you think videos on Youtube still last too long, you can increase or decrease speed with shortcuts shif+. and shift+,, look at this page to a more complete list of shortcuts.

dimanche 7 janvier 2018

First months as a freelancer

I realize I hardly take time to write these days. Actually, I've had a lot to do for the last few months, with my activity as a freelancer to build, the work in my house and my family to take care of. But I'm having a great time!

I quit my salary job last summer to start my own business as a freelancer. I barely had a professional network, just one or two contacts that lead to nothing. So I created a profile on various freelancer platforms. And it worked! Even if what I earn is bellow my goals, I think it's a good start.

Currently, I'm focusing on short missions. I did not want to go full-time at the beginning since I wanted to be present for my family after more than 10 years working a lot. I also wanted to start slowly, because I ended my previous job quite unmotivated, and basically I wondered if I still wanted to do programming for a living.

And I found some missions, I realized the job on time and clients have been quite happy with what I delivered. That's a good point. Here is what I have done so far:

  • A script to save NASDAQ stock events in a database and to perform simple analytics on it,
  • Writing of functional specifications an mock-ups in balsamiq,
  • Add features for a wysiwyg editor for a printing company,
  • Some web scrapers to automatically retrieve invoices from websites.

I'm glad because I worked with Python and JavaScript, which is what I want to become specialized in. It's not much but I learned a lot:

  • I can find missions and get paid.
  • I do great job.
  • I like it.
  • It's not a big deal if I don't get all the missions I apply for. There will be other opportunities.

So to me it's a good start and I hope to work more in 2018, to meet great people, and get decent revenue from this!

So, do I still want to write software for a living? Of course I do.

lundi 20 novembre 2017

Poor man's pomodoro

It seems that I have not written about it until know: I have to confess I'm a big fan of the Pomodoro Technique.

Now that I have a new laptop, I've been looking for a timer software. On Windows and MacOs, I used Tomighty. Cool, but Qt version is not in Debian repo and Java version… well it runs on the JVM!

Then I learned about a magical command, notify-send, which sends desktop notifications. So here we go, my timer will be super simple and written in shell script:

# pomodoro aliases
alias PS="pomodoro_start"
alias PSB="pomodoro_short_break"
alias PLB="pomodoro_long_break"

# launch a 25 minute pomodoro
pomodoro_start() {
    # I write the start time, in case I miss the notification
    echo "pomodoro start `date +%H:%M:%S`..."
    sleep 1500

    # using critical level of urgency force me to click on the notification
    # pop up to dismiss it.
    notify-send  --urgency=critical "Poor man's pomodoro" "Time for a break"

    # cvlc is VLC CLI. I play a sound to help me to get out of the zone.
    cvlc --play-and-exit --quiet /some/sound.mp3 > /dev/null 2>&1

    echo "pomodoro end `date +%H:%M:%S`..."
}


pomodoro_short_break() {
    echo "pomodoro break `date +%H:%M:%S`..."
    sleep 300
    notify-send --urgency=critical "Poor man's pomodoro" "Break over, go back to work"
    cvlc --play-and-exit --quiet /some/sound.mp3 > /dev/null 2>&1
    echo "pomodoro end `date +%H:%M:%S`..."
}


pomodoro_long_break() {
    echo "pomodoro long break `date +%H:%M:%S`..."
    sleep 900
    notify-send --urgency=critical "Poor man's pomodoro" "Long break over, go back to work"
    cvlc --play-and-exit --quiet /some/sound.mp3 > /dev/null 2>&1
    echo "pomodoro end `date +%H:%M:%S`..."
}

The Pomodoro Technique is far to make unanimity. I do not practice it strickly by the book. What I like is that it forces me to get out of the zone when I work on a long running task. Though being the zone and experiencing the flow feels good, relying on my timer helps me to get out of it, take a step back from your work and think. Does what I'm doing right know really worth it?

Contrary to what is usually told about the flow, I don't think it's more productive. It's just pleasant. So while it's a good thing for your hobbies to gain the most of energy out of them, it may not be so appropriate when you have stuff to do at work.

mercredi 11 octobre 2017

Why I chose GNU Linux for my professional laptop

Before you ask the question, or even care about it, here is why I chose to put Debian on my new laptop.

I have to say I really do not like Windows. I'm sure the system got better and better among its versions and that Microsoft itself is improving its opensource policy. However, I do not like how Windows looks, I do not like how it works, I feel in jail with it. In my previous job, my screen used to show a terminal with Cygwin and Vim most of the time, helping me to gain a nice nerd reputation!

The classic dilemma then is to choose between buying a Macbook, or install Linux an another laptop.

We have a 2013 iMac at home and I was globally happy with it at first. Everything is simpler with it, the system is stable, the hardware is beautiful. But with the updates, some stuff got annoying:

  • The OS randomly refuses to mount my USB keys
  • Launchpad randomly refuses to allow me to search apps with the keyboard
  • The system takes longer and longer to boot
  • The system won't launch graphical app (even the Finder) from within tmux

Moreover, last releases of Macbooks are sooo expensive, and lots of people complain about hardware quality on Twitter.

I started regretting my previous experience with Debian on my previous PC. On the other hand, I know that with Linux, I can spend a lot of time trying stuff and fix (sometimes non existent) issues.

So I chose to buy a Lenovo ThinkPad, because I saw at Pycon that a LOT of people had one. Chances are that they are Linux friendly (besides, I think I've read somewhere that they're used by people from Red Hat).

I chose Debian because of the free software philosophy behind it, the large choice of packages and the fact that private software are often distributed as Debian packages. I hesitated with Ubuntu. But seeing Amazon ads on the default desktop on Ubuntu decided me.

I completed the installation two weeks ago and everything goes surprisingly well, far beyond my expectations.

For now, I keep the dual boot with Windows, in case I have to use it for work. But I hope to drop the partition soon!

If you're questioning yourself about going on Linux for your professional laptop, here are some advices.

  • Consider buying a Lenvo Thinkpad. They are well supported and you have a customer support. I read complains about Dell XPS, though they are shipped with Ubuntu. I also saw people with Asus Zenbooks, it can be a good alternative.
  • You'll have software and hardware issues for sure. Yet, during my Linux year I always found a solution eventually on the Internet. In the extreme case, you still can boot on the latest release of the kernel. It's not that hard.
  • You can easily taylor a system that fits your needs and helps you to be more efficient, it's priceless when you rely on your laptop to make money.

samedi 7 octobre 2017

Debian on my new ThinkPad T470p

As I started my new activity as a contractor, I bought a new laptop: a Lenovo Thinkpad T470p. Not that I fell in love with Windows, but to install Debian on it. Here is the description of the journey. I leave the reason of my choice in another post.

Hardware

The hardware is made of:

  • A i7 processor
  • 8 GB Ram
  • 256 GB Samsung SSD
  • A Nvidia GeForce 940 mx graphic card
  • Intel 8265 wifi component

I'm quite happy with this.

What you'll need

Prepare this stuff or it won't work:

  • Your laptop
  • An USB key (not a big one, I used a 1 GB old key I had)
  • An Internet connection through ethernet cable, since what I describe won't work by wifi.

Installation

I have several installation of Linux behind me, but only on BIOS based systems and raspberry PI. So I learned about UEFI, not without some difficulties!

I want to install Debian testing, because Debian stable usually contains old stuff. So at the time I write these lines, its little name is Buster and it might be Debian 10th release.

First thing to do is to download an ISO. Do yourself a favour and don't chose a Debian testing installer. It's an alpha version and will simply not work (I used netinst ISO and it doesn't contains the mandatory packages). Go on with stable distribution. I chose netinst, because I did not have a big bootable USB stick at home and I want a minimal installation. I chose an amd64 architecture (though you have an Intel CPU… I know, I know…).

Now you have to make an UEFI bootable USB key. To do so, from Windows 10, I used Rufus. I first made the mistake of choosing to make it BIOS bootable. The thing is that depending the type you choose (UEFI or BIOS), the installer will now if it has to install on UEFI or BIOS. I made the bad choice at first and the laptop did not boot at the end of the installation. So, choose to format the key as a UEFI boot device.

Remember to shrink your disk to have some room for your Linux system. I used the disk management tool provided with Windows 10. I did not gain lot of disk space, but I guess it is the most secure way to do so.

Now reboot having the key plugged and go into the BIOS by pressing Return while booting. In there, disable Secure Boot in security options (I guess). You can also change boot order to make it boot directly from the disk, instead of Windows bootloader. Exit the BIOS and press F12 to access the list of devices to boot from. Choose the USB stick. You now enter the installer.

From here, you won't face a lot of trouble, do whatever you want. Just notice:

  • I prefer the simple ncursed based installer instead of the graphical one. I know I'm a nerd but believe me, it's easier to use.
  • The wifi chip needs a non free firmware that is not provided by the ISO. The installer propose you to get it from another support. Say no.
  • I advise you to keep the installation to its minimum, without a graphical desktop since you will upgrade to testing right after the process.
  • I chose not to use Swap partition. Come on, I have 8GB RAM! It's controversial though, and I could create a swap file later on if things goes wrong.

At the end of the installation, remove the USB key and let the installer reboot the machine for you. You should then see Grub to boot on your Debian.

Post installation

First thing I did was to switch to testing repositories and activate contrib and non-free repo. To do so, replace the content of /etc/apt/sources.list with:

deb http://ftp.fr.debian.org/debian/ testing main contrib non-free
deb-src http://ftp.fr.debian.org/debian/ testing main contrib non-free

deb http://security.debian.org/debian-security testing/updates main contrib non-free
deb-src http://security.debian.org/debian-security testing/updates main contrib non-free

deb http://ftp.fr.debian.org/debian/ testing-updates main contrib non-free
deb-src http://ftp.fr.debian.org/debian/ testing-updates main contrib non-free

Then, as root, launch the upgrade:

apt-get dist-upgrade

Remember to install the wifi firmware with:

apt-get install firmware-iwlwifi

And voilà, you can install whatever you want.

Troubleshooting

I faced a few caveats after the installation, partly because I chose to use XFCE as my desktop environment and lightdm as a display manager (because I like to try different desktop).

The main problem was the inability to log back to my session after locking it. After spending a looong time on Google, I found that it is due to an incompatibility between XFCE4 power management and lightdm-locker, the locking frontend for lightdm. The no brainer here is to install and configure the (ugly) xscreensaver, as XFCE tries to use it before light-locker while locking the session. Its done in a simple shell script (/usr/bin/xflock4), so feel free to check and hack it if you want.

I also found some error messages related to firmware in the syslog, which made me install firmware-misc-nonfree package. I thought at first it was related to the locking problem, but actually I do not know the effect of this fix.

Conclusion

I'm really happy with this system. It's amazingly fast. I took time to install Node and Python 3.6 from sources and I'm ready to hack!

If everything works well in near future, I may drop Windows partition and reinstall the system to use the whole disk and encrypt the partitions. The journey is far from being over!

vendredi 22 septembre 2017

Adopting vim native package management.

OK, today, I switch to Vim 8 native package management.

Before, I used Pathogen with controversial git submodule feature.

I think it's always better to use native features instead of third party ones, even if I never had any issue with Pathogen.

If you have a similar configuration, just type following commands to switch (assuming your modules are in .vim/bundle directory)

cd ~/.vim  # go to your vim config directory
mkdir -p pack/myplugs/start  # create a pack dir for plugin loading on startup
for DIR in bundle/*; do git mv $DIR pack/myplugs/start; done  # move every submodule from bundle to pack keeping git aware 

You may have to update your git submodules if they emptied in the process:

git submodule update --recursive --remote

Then edit your ~/.vimrc file to remoce call to pathogen

" remove this line
call pathogen#infect() 

Rerun vim, to check if everything is OK for you. Cleaning the .vim directory is left as an exercise.

Native package management feature adds plugins to the runtime path at startup for bundles stored in .vim/pack/whatever/start. You can also configure optional module by putting them in .vim/pack/whatever/opt. If you put a module foodebug in that directory, you have to activate it manually with command:

:packadd foodebug

You could achieve the same with Pathogen though.

Unfortunately, using packages loaded at vim startup does not make their documentation directly available with :h command. To do so, it seems you have to manually update your help tags using:

:helptags ~/.vim/pack