nekudo.com - Blog

ShinyGeoip Newsflash: SSL, Performance, Servers and more

Sun, 04 Mar 2018 00:00:00 +0000

It's been a while since I published some technical information about my Geolocation API. So in this article I want to provide some information about the current state of the projects and talk about some of the problems I had to deal with during the last year (or so).

Server performance

SSL

In February 2016 I first enabled SSL at geoip.nekudo.com. Since then SSL traffic has increased a lot - especially since the rise of LetsEncrypt and the latest Google Chrome updates.

Unfortunately (for this project) an https request uses a lot more CPU time on the webserver than a regular http request. Also the key-length of the certificates private key has a significant impact on server performance. Here you can find a nice article on SSL performance and why it may be a better choice to go for a 2048 bit key if security is not your primary concern.

Meltdown and Spectre

Then of course there where Meltdown and Spectre which may or may not have an impact on your servers CPU usage. It probably depends on your setup but it's not totally unlikely that there is also some increase in CPU usage caused by this patches.

Traffic report

The geoip.nekudo.com API currently runs on two servers with an identical setup. Here are the traffic graph showing the overall traffic on the eth0 interface as well as the requests per second hitting the Nginx webserver during the last 24 hours.

Since the traffic is distributed evenly between the two servers the graphs from the second server look more or less the same.

As you can see the API currently handles a daily average of about 600 req/s per second.

Traffic peaks

Of course there will always be traffic peaks like when a new big websites experiments with the API or one of the websites using the API experiences a DDoS attack. This is why I always try to have sufficient resources to handle a least double the usual traffic.

Server updates

With a general increase in traffic as well as an increase in SSL traffic in special it is not surprising that I needed to update the severs a few times during the last year(s).

Most recently I switched from 3 smaller severs - to two severs with more CPU power to be able to serve all requests as quickly as possible. I run the API on at least two servers in different datacenters at all times. So in case of a problem I can keep downtimes to a minimum.

Of course all sever-updates are (generally ;)) applied without any downtime at all.

Donations

Last but not least I added a "Donate" link to the ShinyGeoip project page. In case you want to help to pay for the servers feel free to consider a small donation. Of course the usage of the API will remain free and without any limitations either way.

If you have any additional questions feel free to contact me.

]]>

Using LetsEncrypt in a multi-server environment

Sun, 05 Feb 2017 00:00:00 +0000

There are plenty of tutorials out there covering the default LetsEncrypt setup for only a few domains running on a single server - but when it comes to more special setups including multiple servers there is not much information. This is my solution for using LetsEncrypt within a multi-server setup.

The problem

Most probably my setup is relatively common: I host multiple projects on multiple servers using multiple domains. One kind of special case is a round-robin DNS setup where the traffic is distributed to the servers using multiple A-records in the domains DNS record.

When using one of the LetsEncrypt ACME Clients you will probably face some problems - at least I did ;).

Problem 1: Dependencies

Depending on which ACME client implementation you chose it has various dependencies like e.g. Python, Node.js, ... If you use multiple server you most probably don't want to install those dependencies on each of your machines.

Problem 2: ACME domain validation

When generating an SSL certificates using the ACME protocol you have to verify your domain(s). This is in most cases done using a challenge-response mechanism during which a file containing a token is placed on your webserver which is than requested by LetsEncrypt via a HTTP request. E.g. if you want to create an SSL certificate for the domain example.com LetsEncrypt will do an http-request to: http://example.com/.well-known/acme-challenge/some-random-file

The problem with that method is that most clients only place the authorization file on the server they currently run on - which means you will get in trouble as soon as you use multiple servers.

An additional problem occurs in some load-balancing cases - like e.g. my round-robin DNS setup. In this case you will never know which server will handle the request.

My solution

Just a heads up: This solution works for me. It does not mean is perfect or will work for you.

I decided to use only one central server which handles the certificate generation for all domains and servers. This way I only need to maintain an ACME client on one machine. Additionally I only need to install dependencies (if any) on this one machine.

Redirecting domain validation requests

Using only one server to create SSL certificates means this server needs to handle all the domain-validation requests. Luckily the ACME protocol seems to accept 302 redirects. So I simply created a subdomain (e.g. acme.yourdomain.com) pointing to the SSL handling server and redirected all validation-requests from the various projects to this subdomain using a simple nginx rule on each webserver:

rewrite ^/.well-known/acme-challenge/(.*)$ http://acme.yourdomain.com/$1 redirect;

After this is was done I could generate SSL certificates for all my domains and servers from one machine and than distribute the certificates to their destination machines using simple SCP. Of course you could even write a simple script automating the distribution process - which will be my next step.

Hopefully this short article can prevent some of you from slamming your head onto the keyboard ;)

]]>

Moving from Nginx to aerys - an experiment

Sun, 15 Jan 2017 00:00:00 +0000

In this short article I want to share my experiences with the aerys application framework. I experimentally used the aerys server at my ShinyGeoip instance over at geoip.nekudo.com.

Introduction

The free geolocation API at geoip.nekudo.com currently runs on two servers. Traffic and load is distributed using simple round-robin DNS. Each server handles between 100 and 150 request per second which you can see in the two graphs below.

As this API is totally free of charge I of course try to reduce load on the severs as much as possible to keep the costs low. Because the nginx and php-fpm configuration as well as the PHP code is more or less optimized as much as possible I decided to try something new.

PHP application frameworks

In various blog articles (and PHP events) I read about PHP application frameworks which can replace classic webservers like apache or nginx. These frameworks include there own webserver and are completely written in PHP. Most of them are asynchronous, multithreaded and include a wide range of features like e.g.: a webserver, websocket-server, ssl and http2 support and so on. A few of the most interesting frameworks seem to be:

There are may more similar projects. If you are interested in this topic I recommend Asynchronous PHP which is a collection of interesting projects related to asynchronous programming in PHP.

Switching ShinyGeoip to aerys

I decided to give aerys a shot - simply cause it seems to be actively maintained and is relatively easy to start with.

Converting a small project like ShinyGeoip to the application server was not that hard and you can see the result at this Github repository if you are interested.

The most important thing when switching from a classic webserver to a PHP based application server is to realize that your application does not get initialized with every request but only once when the server is started. After that it only needs to handle the incoming requests. This of course can save lots of resources.

Unfortunately in my experiment I could not improve the performance of the ShinyGeoip API using aerys. I manged to let the application handle the same amount of requests than the nginx server while not increasing the CPU load too much - but the aerys sever consumed much more memory. This resulted in an overall higher load on the server handling the same amount of requests.

Conclusion

Of course this was only my first experiment using a PHP based application server and I can imagine multiple reasons why I did not perform as good as my classical nginx/php-fpm setup:

Maybe there is a memory leak in my application.
Maybe there is a memory leak in the aerys application.
Maybe the use-case is to "special" to benefit from the advantages of a PHP based application server. After all the ShinyGeoip application is relatively small but handles a lot of single requests. I could imagine a much bigger application is more suitable for this kind of server as the overhead of initializing a large application with every request can be reduced much more efficiently in this case.

But all in all I must admit that this kind of application is really interesting and I was surprised how powerful and stable those application server already are. I will definitely give it another try with another kind of application.

]]>

Building microservice based PHP applications using Angela

Tue, 10 Jan 2017 00:00:00 +0000

In this article I will explain how you can use the Angela worker-framework to build a microservice application in just a few simple steps.

What is Angela?

In short: Angela is a PHP framework to build and run worker processes. It provides a job server to manage your workers as well as a client to send jobs from your application to those workers. You can read more about the framework itself on the Angela Github page.

Why would I need microservices?

Microservices are a great way to separate big monolithic application into multiple independent smaller services. This approach has various benefits like e.g. maintainability, reliability, parallelization and much more. If you are interested in the general concept of microservices there are lots of great articles out there.

But it is not necessary to refactor your whole application. You can also use worker-processes for simple tasks like sending out e-mails, processing images and so on. Task you farm out to a worker can run asynchronously and therefore massively speed up you project.

Requirements

Before you can use the Angela framework you'll need a few things. Most important is a webserver which you can access using SSH as the job-server and worker-processes need to be run in CLI mode.

Besides that you'll need PHP 7 with the following extensions installed:

ZMQ PHP extension
ev PHP extension (optional but recommended for better performance)

Installing Angela

The recommended way to install Angela is using composer:

composer require nekudo/angela

Setting up your application

After installing Angela you will find a folder "vendor" in you project root which contains all libraries required to use the Angela framework. Angela comes with a few example files which can be found in "vendor/nekudo/angela/example". To start a new application I recommend to copy this folder into your project root and rename it to "app". This folder will be the base of you new application.

Adjust the configuration

The first step when developing a new Angela based application is to adjust the configuration. To do this open the file called "config.php" in your app folder. All the configuration values are explained inside the file. In most cases the default configuration should work. The only part you need to adjust is the part defining your worker-pool. It looks like this:

'pool' => [        
    'pool_a' => [            
        'worker_file' => __DIR__ . '/worker/worker_a.php',            
        'cp_start' => 3,
    ],
],

In this case there is one pool defined named "pool_a". The worker for this pool is named "worker_a.php" and will be started 3 times when you fire up the server. This means once you start the job-server there will be 3 workers of type worker_a and each of this workers can do one task at the same time. So if you have lots of tasks which can be handled by worker_a you can just start more workers and jobs can be processed faster.

Writing your first worker

After you configured the worker-processes you want to use you'll need to actually implement those workers.

For each worker you defined in the configuration there needs to be a corresponding file containing the code. In this example the worker can be found in your app/worker folder and is called worker_a.php. The content of a very basic worker looks like this:

<?php
namespace Nekudo\Angela\Example;
use Nekudo\Angela\Worker;
require_once __DIR__ . '/../../vendor/autoload.php';

class WorkerA extends Worker
{
    public function taskA(string $payload) : string
    {
        // Do some work:
        usleep((rand(2, 5) * 100000));

        // Return a response (needs to be string!):
        return $payload . '_completed_by_' . $this->workerId;
    }
}
// Create new worker and register jobs:
$worker = new WorkerA;
$worker->registerJob('taskA', [$worker, 'taskA']);
$worker->run();

Each worker needs to extend the Angela Worker class and should contain at least one job-type it can handle.

In this example the worker can handle one type of job called taskA which does nothing more than sleep for a short period of time and than return the payload you sent to the worker and its own id. In a real world application these files would contain the code to do the actual work you want to farm out of you main application - like e.g. resizing images, sending emails, crawling feeds and so on.

Starting the job server

After you implemented you first worker you can now fire up the job-server using the following command inside your app folder:

php control.php start

If everything went okay your server and the worker processes are now running and waiting to complete jobs. You can check the status of your server by typing:

php control.php status

This should output something like:

------------------------------ SERVER STATUS ------------------------------
Angela Version: 2.0.0
Start time:     2017-01-10 20:24:41
Uptime:         0d 00h 03m 02s
------------------------- WORKER/JOB INFORMATION --------------------------
Job requests total:   0
Current queue length: 0

Active worker per pool:
 + pool_a: 3 

Jobs completed per worker:
 + s1_6760: 0
 + s1_6761: 0
 + s1_6762: 0

Passing jobs to a worker

When your server is up and running you can now pass jobs from you application to the job-server by using the Angela Client:

<?php
require_once __DIR__ . '/../vendor/autoload.php';

$client = new \Nekudo\Angela\Client;
$client->addServer('tcp://127.0.0.1:5551');
$result = $client->doNormal('taskA', 'some workload');
$client->close();

This sort example inits the Angela client, connects to the job-server and sends one job-request of type "taskA" to the server. It than waits for the result of the job. As soon as a worker has processed this job, the worker sends back the result of the job to the job-server which than forwards this result back to the client.

This example of course would block your application until the result of the job is received. But in many cases you don't need an actual result - e.g. if you just want to resize an uploaded image to some predefined sizes. In this case you can use the doBackground method:

<?php
require_once __DIR__ . '/../vendor/autoload.php';

$client = new \Nekudo\Angela\Client;
$client->addServer('tcp://127.0.0.1:5551');
$jobId = $client->doBackground('taskA', 'some workload');
$client->close();

This code will trigger the same job-request as the previous code but it will not wait until the job is completed. You will just receive a job-id and your application can continue to process code right away. This is extremely useful if you have some operations running relatively long and you don't want your users to wait for the page to load.

Conclusion

I hope this short example could explain that you can farm out work from you main application into other (background) processes in a few easy steps. Of course it's always hard to adopt this pattern into an existing application but you should keep it in mind for you next relaunch ;)

]]>

ShinyGeoip Version 2.0 - A performance update

Sun, 10 Apr 2016 00:00:00 +0000

With increasing traffic on my free GeoIp API I decided to take a detailed look onto the application powering the API to check if performance improvements were possible. In the end the application got the whole nekudo performance overhauling. This is a detailed report about what I changed.

Analyzing the status quo

The first thing I do when analyzing performance of a web application is to check which are the most common requests hitting the server. In this case this was easy - almost every request is requesting location data from the REST API like e.g.:

http://geoip.nekudo.com/api/195.14.223.202

So I did one of those requests on my local machine and recorded a profile using Xdebug. I highly recommend to do such a profiling as it is an easy way to find out what exactly is going in on your application. This is the result:

The result was kind of shocking: Hundreds of method calls for an extremely simple task. And if you take a closer look you will notice that most of the executed code is not even from the core application but from the Slim framework, composer and other vendor libs.

Making a plan

After analyzing the results I defined some goals for the version 2 sourcecode:

Drop Slim (or any other) framework.
Drop composer and autoloading.
Drop as many vendor libs as possible.
Reduce complexity of sourcecode to as little methods as possible while still keeping a clean structured codebase.
Do not change API or API responses.

The last point was very important due to the fact that that there are many websites using the API of http://geoip.nekudo.com and I did not want to break their code.

The result

I managed to adjust the sourcecode and achieve all the goals. I dropped Slim and Composer and cut the used vendor libs down to one. This one luckily is available as PHP extension implemented in C so it's extremely fast. I also refactored the sourcecode of the core application while sticking to the Action-Domain-Responder pattern.

Here is a profiling of the same request as before hitting the version 2.0 source:

The direct comparison is quiet impressive. There is now only about 30 method calls before delivering the result to the user. The overall time cost is only about 4% of the time the same request took in version 1.

Of course: This is only a few microseconds but on a "high traffic" website this can make a huge difference as I will show in the next section.

The deployment

My free geolocation API at http://geoip.nekudo.com currently runs on two servers. One machine runs PHP 5.6 the other one is on PHP 7.0.5. I deployed version 2.0 of ShinyGeoip to both servers today - here is what happened:

The two graphs show the CPU usage and requests per second hitting the nginx server. You can clearly see how the CPU usage goes down after deploying version 2.0 of the software while the req/s stay at the same rate or even increase.

This was the PHP 5 machine. Here are the same graphs for the PHP 7 machine:

It's basically the same result but it seems the effect is slightly higher on PHP 5 than on PHP 7.

Conclusion

My lesson learned: Optimizing your source-code matters even more on "high traffic" websites. I definitely recommend to do a sourcecode-profiling of you application to reveal possibilities of performance improvements.

And (I can't say this enough): It's not always the best solution do develop an application by selecting a framework and pull in some vendor libraries using composer. I know it's tempting and fast but be careful when it comes to performance.

Happy hacking :)

]]>

ShinyBlog - A markdown based Blog/CMS

Sun, 27 Mar 2016 00:00:00 +0000

A few days ago I completed my work on a new project and it is now available for download at GitHub. ShinyBlog is a lightweight Blog/CMS application entirely based on markdown files. Of course the software is free and open-source.

Motivation

Until recently this website was based on TextPress which also is a markdown based blog engine. Unfortunately the software does not seem to be maintained any longer and was missing some features so I decided to build an alternative. Of course there is lots of other Blog/CMS applications out there like. e.g Wordpress or Grav. But most of these applications are much to bloated for a simple blog like this one.

This is why ShinyBlog is super lightweight. It was developed with simplicity in mind. There is no composer, no install-script and very few vendor-libraries were used. This way the code stays extremely simple and clean which makes the software extremely low maintenance and secure. It (hopefully) simply works. If not: Create an issue at GitHub ;)

Features

Here are some of the features I implemented:

Markdown based

The whole system is based on markdown flatfiles. So adding a new artcile to your blog is nothing more than uploading a new textfile to your server. You can even use your existing deployment system if you like.

Theme support

The application includes a very basic theme but you can easily build your own. Just copy the existing "kiss" theme to a new folder and adjust the CSS and/or HTML. Themes can be swiched using a simple config file.

RSS Feeds

All articles can be displayed within an RSS feed. Additionally there are RSS feeds for each article category. This way visitors can filter articles of a special category in there RSS reader.

SEO support

There are some basic SEO settings available. You can define title and meta-descriptions for each article/page. Additionally you can define noindex/follow stuff via config file.

Misc

Pagination: Set how many articles you want to show per page.
XML sitemap: Sitemap of all articles and pages is automatically generated.
Excerpts: Using a read-more tag in your articles you can define the excerpt to appear on the blog-page.
Clean Code: Well documented, PSR-0 and PSR-2 compatible PHP code.

So if you are interested in an extremely simple blog engine like you see it right now on this website you should check out ShinyBlog. Have Fun.

]]>

Optimizing MySQL performance using bitmasks

Sun, 10 Jan 2016 00:00:00 +0000

When it comes to (mysql) database optimization the most common practice is to normalize your database structure. In general this is a good idea but when it comes to performance a completely normalized database is not always the best solution.

For a better understanding of the following text I will use the following example data:

Table shirts
+----+----------+
| id | motive   |
+----+----------+
|  1 | Homer    |
|  2 | Marge    |
|  3 | Bart     |
+----+----------+

Table colors
+----+----------+
| id | color    |
+----+----------+
|  1 | Red      |
|  2 | Blue     |
|  4 | Green    |
|  8 | Black    |
+----+----------+

We have a table containing t-shirts and another table containing the colors. Every shirt can be available in different colors - but not every shirt is guaranteed to be available in every color. So what we need to do is to create a new table to store the information which shirt is available in which color. This would probably look like this:

Table: shirts_x_colors
+----------+------------+
| shirt_id | color_id   |
+----------+------------+
|        1 |          1 |
|        1 |          4 |
|        2 |          2 |
|        2 |          8 |
|        3 |          2 |
+----------+------------+

This table now tells us which shirt is available in which color - like e.g. shirt "Marge" is available in red and black. So if we would like to have a list of all shirts and available colors we would need to query 3 tables and probably even use a GROUP_CONCAT to get a comma-separated list of the colors. Something like:

SELECT shirts.motive, GROUP_CONCAT(colors.color)
FROM shirts
JOIN shirts_x_colors ON shirts.id = shirts_x_colors.shirt_id
JOIN colors ON shirts_x_colors.color_id = colors.id
GROUP BY shirts.id

As you can imagine this type of queries can become relatively slow because your shirts probably not only have a color but also different sizes, materials, ...

Using bitmasks

Bitmasks can help to improve this kind of queries. As you probably noticed the IDs within the colors table are not just simply an auto-incremented integer but only numbers which are powers of two (1,2,4,8,16,32,...). We can now extend our shirts table like this:

Table shirts
+----+----------+--------+
| id | motive   | colors |
+----+----------+--------+
|  1 | Homer    |      5 |
|  2 | Marge    |     10 |
|  3 | Bart     |      2 |
+----+----------+--------+

I simply added a new column called "colors" which contains the sum of all available color-ids for each shirt - e.g: Shirt "Homer" is available in Red (1) and Green (4) so "colors" is 1 + 4 = 5. The cool thing about bitmasks is that this sum is always unique - meaning only from this sum we know in which colors a shirt is available.

Here is some sample code converting the sum of our ids back into the single values:


function reverseBitmask($bitmask)
{
    $bin = decbin($bitmask);
    $total = strlen($bin);
    $stock = [];
    for ($i = 0; $i < $total; $i++) {
        if ($bin{$i} != 0) {
            $bin_2 = str_pad($bin{$i}, $total - $i, 0);
            array_push($stock, bindec($bin_2));
        }
    }
    return $stock;
}

print_r(reverseBitmask(5));

But there's more: What if we would like to search for all shirts available in color red or green? We can do this with one simple query. The sum of red and green ids would be 5 so our query looks like this:


SELECT * FROM shirts WHERE colors & 5 > 0;

This would return a list of all shirts available in either red or green.

But be careful: The query above can not make use of a MySQL index. In terms of performance it sometimes may be better to use a simple IN(x,y,z) query. (Thx to Alan for this hint!)

Conclusion

Using bitmasks can help you to improve the performance of your SQL queries in some cases. Of course you have to carefully check if this technique is useful in your case. For example I would not suggest using bitmask if the list of attributes (e.g the colors) can become very long. Additionally you have to be clear that you will store redundant data in your database if you change the structure of your tables according to the above example.

Using bitmasks is no magic-bullet to solve database performance issues but it can help in some cases - and I hope now you have a basic understanding of how bitmasks work and can maybe use them in your next project.

Happy coding!

]]>

ShinyDeploy - A deployment tool for web applications

Wed, 09 Dec 2015 00:00:00 +0000

In this article I want to introduce "ShinyDeploy" - a project I have been working on during the last month. As many features are already implemented and working I decided that it's time to introduce this application to the community and hopefully find some beta-testers.

Introducing ShinyDeploy

ShinyDeploy is a deployment tool for web applications written in PHP and JavaScript. It's supposed to provide an easy way to deploy files from your GIT repository like GitHub or Bitbucket to your webserver.

You can find the installation instructions at the projects GitHub page.

Here is a list of the features and some explanatory texts:

Basic and straightforward GUI

The application uses a clean and simple user-interface enabling everybody in your team to quickly deploy changes to your servers.

List changes before deploying

If you're not sure about all the changes since the last deploy or you just want to double-check stuff before hitting the deploy button - you can list all modified files and even view a diff before starting the actual deployment.

Execute tasks before/after deployment

For every deployment you can define commands that are executed before or after you deploy changes to the server. This can be stuff like putting your application into maintenance mode, compiling SCSS files after upload, restating software and so on.

Realtime feedback using websockets

Using websockets you get direct feedback of what's happens during a deployment. This is especially helpful in situations when something is not working as expected ;)

Webhook support

If you like automation you can use webhooks (like provided by GitHub or Bitbucket) to automatically trigger deployments. Using webhooks whenever you push changes to you repository the will automatically get uploaded to your server. This is an extremely useful feature to keep your development server up to date.

Open source and self-hosted

Using a deployment tool automatically means you have to provide sensitive information (like login data to your servers) to this tool. I personally don't like to provide such information to external services having no insights in how they handle these data. Using open source self-hosted software means you maintain full control of your data.

Secure data storage

All sensitive data like usernames, passwords e.g. are encrypted before storing them into database. This provides an additional layer of security. If an attacker manages to get a copy of your database the data is still encrypted and can't be used.

Not too hard to setup

I tried to keep the requirements and installation procedure as simple as possible. Sure it's not like installing Wordpress but with a little experience in server administration it should be possible.

Disclaimer

Please be aware that at moment the project is in an early beta stage. It has not been tested by a wide range of users and most probably still contains some bugs. I use the application at daily basis but for now I don't recommend to use it in production. Just test it on your development or staging server.

Contribute!

If you're interested in testing and improving this application you're very welcome. Just start testing and provide some feedback. Open issues at GitHub or write me a mail. I'm excited to get some feedback.

]]>

Introducing: Angela - A gearman worker framework

Sun, 08 Nov 2015 00:00:00 +0000

In my last post on gearman microservices I already mentioned the code I am using to manage Gearman workers in my applications. Now I have isolated the code from the rest of the project and created a simple framework called Angela. It contains all the necessary methods to create and manage Gearman workers in a PHP application.

The sourcecode is available at GitHub including a demo folder which contains a simple dummy application to illustrate the usage of this framework.

Feel free to check it out, contribute or submit issues.

Btw: Why Angela? Cause Angela is the boss ;)

]]>

Autoloading performance while using Composer

Thu, 15 Oct 2015 00:00:00 +0000

Almost every PHP project these days uses composer to manage its dependencies. One of the advantages is that it manages the class loading for you. You just need to use the correct namespaces in your sourcecode and the classes will be automatically loaded. But what about performance?

Default composer class loading

To understand the "problem" with performance when using composer we need to understand what autoloading means. I won't get to much into the details cause everybody who is interested can just have a look into the composer source. It not really hard to understand. In short: Composer analyzes the namespace of a class and then checks if the class exists at some places. If the class is found it is included.

That looks like this:


private function findFileWithExtension($class, $ext)
    {
        // PSR-4 lookup
        $logicalPathPsr4 = strtr($class, '\\', DIRECTORY_SEPARATOR) . $ext;

        $first = $class[0];
        if (isset($this->prefixLengthsPsr4[$first])) {
            foreach ($this->prefixLengthsPsr4[$first] as $prefix => $length) {
                if (0 === strpos($class, $prefix)) {
                    foreach ($this->prefixDirsPsr4[$prefix] as $dir) {
                        if (file_exists($file = $dir . DIRECTORY_SEPARATOR . substr($logicalPathPsr4, $length))) {
                            return $file;
                        }
                    }
                }
            }
        }

And here you can see a little problem. As it is not always unambiguously defined where to find a file - the file needs to be "searched". At this point we waste some time cause it is of cause possible to cleary define where to find each file.

And fortunately composer offers a solution for this!

Using optimzed autoloading

After you installed or update something using composer just type:

composer dump-autoload -o

You can find the detailed explanation of this command in the composer docs.

In short: It creates a class-map defining the path on you server for every possible namespace. This looks like this:


return array(
    'Guzzle\\Batch\\AbstractBatchDecorator' => $vendorDir . '/guzzle/guzzle/src/Guzzle/Batch/AbstractBatchDecorator.php',
    'Guzzle\\Batch\\Batch' => $vendorDir . '/guzzle/guzzle/src/Guzzle/Batch/Batch.php',
    ...

Using this method we don't need to search for files any longer and should improve performance.

Performance analysis

Of course I wanted to know if it really makes a difference to use the class-map so I tested this using the ShinyGeoip project as an example.

The first image shows a profiling of an API request to the ShinyGeoip API whith composer in normal mode. The method findFileWithExtension is called 41 times and consumes 19 percent of the time during this request. As the whole project is relatively small 19% is quite a lot and it would be nice if we could save some time here.

The second image shows the same request but with composer using the classmap-file. The findFileWithExtension method is not called and at the same point we're already a few steps further processing the code.

Some numbers

Additionally I crunched some numbers to see if there is any measurable difference when calling the API. So I called the API 20.000 times in blocks of 1000 requests an measured the times. Here are the results:

6.9331228733063 sec
6.0901930332184 sec
6.2068719863892 sec
5.9258079528809 sec
7.7793209552765 sec
7.2444310188293 sec
7.3575570583344 sec
7.2506470680237 sec
7.5955770015717 sec
6.7635560035706 sec
7.3395359516144 sec
6.5393271446228 sec
6.1258180141449 sec
8.2368841171265 sec
5.5511469841003 sec
4.5876572132111 sec
7.8379991054535 sec
6.3960349559784 sec
6.5366761684418 sec
6.5052518844604 sec
Avg. time 6.7401708245277 sec

1000 requests with composer in normal mode took about 6.7 seconds in average on my local machine.

7.1470701694489 sec
5.9239280223846 sec
6.5201210975647 sec
6.8260641098022 sec
5.9228749275208 sec
5.9340260028839 sec
6.3986730575562 sec
6.1228239536285 sec
6.1717998981476 sec
5.8733088970184 sec
5.7348520755768 sec
6.3911769390106 sec
6.1153709888458 sec
6.3705530166626 sec
5.092787027359 sec
6.2643351554871 sec
6.2573671340942 sec
6.007838010788 sec
6.7526230812073 sec
6.3487119674683 sec
Avg. time 6.2088152766228 sec

With composer using the classmap-file the same 1000 request took about 6.2 seconds in average. So there is a measurable difference using this improvement.

Final thoughts

As always when doing such a performance analysis: This is micro-optimization and you should only invest time in such improvements if you can afford it. But my primary goal is to raise the awareness of these little things. Don't just use tools as they are - try to understand and improve them. And of course: It's always good to have a look into the documentation ;)

]]>

ShinyGeoip project status

Sun, 11 Oct 2015 00:00:00 +0000

About 10 month ago I released ShinyGeoip and started the free geolocation API at geoip.nekudo.com so I guess it's about time for a quick status update.

The code seems to be quite stable as I had to fix only minor bugs during the last 10 month. For performance reasons I decided to kick the Basscss toolkit and replace with some own lines of CSS to reduce the overhead when loading the homepage.

API statistics

In the graph you can see the Nginx requests per second from the last month. The API currently handles about 5-10 requests per second in average. That's about 600.000 requests per day. There's still room for more ;)

During peaks there are 250-500 requests per second. But I recognized that these peeks mostly request the same IP over and over again. This most probably happens when a site using the API get's brute-forced by a bot. To handle this the nginx configuration only allows 30req/sec from one IP.

So all in all the project runs very well. Let's see where it is going.

]]>

Microservices in PHP using Gearman

Thu, 08 Oct 2015 00:00:00 +0000

As microservices seem to become more and more popular right now I decided to write an article about Gearman. Why? Because in my opinion Gearman is a great tool to implement the microservices pattern into PHP applications. Additionally I'm using Gearman in my projects for quite a while now. But let's start from the beginning.

What is Gearman?

Gearman is a framework which allows us to separate single tasks from our application and run them separately - loosly coupled from the rest of the code. In the microservices pattern such a task would be called a service. Gearman calls it a worker.

On the other side we have a client. The client creates jobs which are then handled by our workers. When implementing the microservices pattern the client would probably live in the "gateway" or "application gateway".

Between client and workers we have a job server. The job server basically handles the communication between client and worker. Every worker is registered at the job server. When the client passes a new job to the job server - it will find a suitable worker and hand over the job. The worker will then try to finish the job and eventually pass back data to the client via the job server.

If you need more information on how this works please have a look onto the Gearman homepage which explains the framework very well.

Why should we use Gearman?

When we start to implement the microservices pattern in our application we need to find solutions to a few tricky problems. For example: We need a way our application can communicate with out services. This is often done using a simple HTTP REST API. But what if we want multiple instances of our services working in parallel? What if we want to distribute the services accross multiple servers? Or even implement a service in another language because it seems more suitable for the given task? The Gearman Job Server already provides all this features.

Using Gearman we can start as many worker accross as many servers as we like. This makes our application extremely scalable. If our application generates lots of task for a special service - we just start more workers handling this task. If our workers consume lots of resources - we just add a new server and fire up some more workers.

Gearman actually offers interfaces for a lot of languages like PHP, C, Go, .NET, Java, NodeJS, ... So we are able to implement the various services of our application in different languages.

The basics

Starting to use Gearman in a PHP application is extremely easy. After installing all required dependencies (typically this would be the Gearman job server and the Gearman PHP extension) we can start to implement our first worker/service:


class WorkerExample
{
    private $gearmanWorker;

    public function  __construct()
    {
        $this->gearmanWorker = new GearmanWorker;
        $this->gearmanWorker->addServer('127.0.0.1', 4730);
        $this->startup();
    }

    private function startup()
    {
        $this->gearmanWorker->addFunction('foobar', [$this, 'foobar']);
        while($this->gearmanWorker->work());
    }

    public function foobar($Job)
    {
        $payload = json_decode($Job->workload());

        // Do some actual work here...
        var_dump($payload);
    }
}

$Worker = new WorkerExample;

No magic here. We just create a new instance of the GearmanWorker class - which is provided by the PHP extension. Then we register a new function called "foobar" and wait for new jobs. In this dummy the only thing this function would do is echoing out the parameters we passed.

This is how we can create a new job to be processed by this worker:


$client = new GearmanClient;
$client->addServer('127.0.0.1', 4730);
$payload['foo'] = 'Hello Service!';
$payload = json_encode($payload);
$client->doBackground('foobar', $payload);
echo "done."

We create a new instance of the Gearman client and register it at the job server. Than we pass some dummy date to a worker and start it asynchronously - which means: We won't wait for the job to finish. The string "done." would be displayed directly after the job was started.

For more code examples and a list of all the different Gearman features please have a look into the official documentation at php.net.

Pitfalls and possible solutions

As every software Gearman has some pitfalls or things you have to get used to. I'll try to mention an few problems I ran into during my work with Gearman and explain how I solved them. I hope in this part of the article I can even address some users who already have some experience with Gearman.

Addressing special workers

One of the first things we will notice when starting multiple instances of a worker is that there is (by default) no way to address a special worker. Let's assume we have a worker responsible for creating thumbs in various formats from a given image. If we fire up 3 workers of this type there is (by design) no way to address one of these three workers explicitly. As soon as we create a new "resize job" there is no way to control which of the 3 workers is going to do the job. This behaviour is by design and if there is not good reason we should no change it!

But in some cases it is useful to address one explicit worker (for example to check if the process is still responding) and here is how I achieve this:


class NamedWorkerExample
{
    private $gearmanWorker;
    private $workerName;

    public function  __construct($workerName)
    {
        $this->workerName = $workerName;
        $this->gearmanWorker = new GearmanWorker;
        $this->gearmanWorker->addServer('127.0.0.1', 4730);
        $this->startup();
    }

    private function startup()
    {
        $this->gearmanWorker->addFunction('foobar', [$this, 'foobar']);

        // This does the trick:
        $this->GearmanWorker->addFunction('ping_' . $this->workerName, [$this, 'ping']);

        while($this->gearmanWorker->work());
    }

    public function foobar($Job)
    {
        $payload = json_decode($Job->workload());

        // Do some actual work here...
        var_dump($payload);
    }

    public function ping($Job)
    {
        $Job->sendData('pong');
    }

    $Worker = new NamedWorkerExample('w1');
}

I used the same example code like above and added a new method "ping". Additionally a worker name or id is passed into the worker when creating a new instance. This name is appended to the function name when we register the ping method as callback to the Gearman server - this is what does the trick. If we now fire up multiple instances of this worker - but every one with a different name - we are able to address every worker instance separately. Here is an example:


$client = new GearmanClient;
$client->addServer('127.0.0.1', 4730);
$start = microtime(true);
$pong = $client->doHigh('ping_w1', 'ping');
$pingtime = microtime(true) - $start;

This will explicitly "call" the ping method in the worker with the name "w1".

Monitoring workers

One of the trickiest parts when working with Gearman is monitoring the workers. When implementing long running scripts in PHP we can be almost sure they will die at some point. First thing that comes to mind most probably is: Why don't we just ping the workers periodically and kill/restart the processes that do not respond?

With the possibility to address special workers like mentioned above this seems like a good idea - but unfortunately it won't work. Here is why: Every worker can only execute one job at a time. Let's assume our workers main job is the resizing of images. Additionally we implement a ping method. Every 5 minutes we ping each of our workers to see if it is still responding - if not we kill and restart the process. But what if the worker is just doing a regular job? It would reply to the ping request - but not before it is done resizing the images. So if we would set our ping timeout to e.g. 5 seconds but the resizing of an image would take 10 seconds our script would kill the worker process although it was totally okay. So this is not a good solution.

Here is my solution for monitoring worker processes:

I always start more workers than actually needed. So if one process dies there are still some other workers which can process the jobs and the application remains functional.
Every worker creates a PID file when started. This is just a file which can explicitly be assigned to the process by e.g. using the worker name as filename.
Worker processes periodically write the current timestamp to this pid file.
I use a cronjob to regularly check the pid files. If a timestamp is greater then a defined timeout the corresponding process is killed and restarted. It is important that the timeout is greater then the jobs usually take so we can make sure not to kill processes that are busy but still responding.

Hint: There will be code samples implementing this solution in the next part of this article.

Managing worker processes

As soon as our projects grows and requires more than 1-2 workers we need to think about a way to manage the worker processes. We don't want to start, restart or stop them one by one on the console.

Unfortunately I can't provide PHP code which will work out out the box with every project but nevertheless I will at least present and explain my current solution. It consists of two parts:

A simple cli script to start/stop worker processes. (Sourcecode at GitHub)
A WorkerManager class with contain the actual code to mange the processes. (Sourcecode at GitHub)

In a config file I define how many workers of each type/service I want to start:

'workerScripts' => [
    'deployer' => [
        'filename' => 'worker.deployer.php',
        'instances' => 3,
    ],
    'repoactions' => [
        'filename' => 'worker.repo_actions.php',
        'instances' => 3,
    ],
]

For example this configuration would start 3 "deployer" and 3 "repo_action" workers.

In the manager I mainly implemented 3 methods which can start, stop and "keepalive" the worker processes. Start and stop should be self-explanatory. The keepalive method is periodically triggered by a cronjob. It checks if the workers are alive and restarts them if necessary using the monitoring methods described above.

Conclusion

In my opinion Gearman is a powerful tool to implement the mircoservies pattern into a PHP application. It allows us to pass jobs from our application to the single services or workers. Additionally provides an easy solution when it comes to scaling. We can just fire up new services across multiple servers without changing a single line of code.

I definitely recommend to give it a try! Happy hacking.

]]>

Wemahu - Project discontinued

Sun, 04 Oct 2015 00:00:00 +0000

About two years ago I started the Wemahu project which is basically a malware scanner for Joomla and Wordpress. Unfortunately this project never really got any attention which has different reasons.

At first I simply did not put much afford in marketing the extensions. Second: It takes much time to keep the signature database up to date. But the third (and probably most important) reason: To do effective malware scanning you will need long running PHP scripts to be executed on the server. Sadly this is often restricted by the hosting companies mainly used for wordpress or joomla installations. The alternative would be to use CLI scripts which is often has the same limitation and additionally overburdens lots of users.

Wemahu has not received any updates in month so I decided to discontinue the project. If you're still interested in using the extensions - or maybe even want to fork them - you can find them in my GitHub account.

For historical reasons here's the extensions changelog:


**Version 1.0.3 (2014-08-23)**

* Whitelist updates.
* Url changes.

**Version 1.0.2 (2014-02-23)**

* Added possibility to exclude folders from file modification scan.
* Minor layout improvements.
* Updated signature and whitelist database.

**Version 1.0.1 (2014-01-25)**

* Bugfix in filestack creation.
* Bugfix: Max. results in rulesets not saved.

**Version 1.0.0 (2013-11-24)**

* First stable release.
* Feature: Option to set if report-mail is send on empty report.

**Version 0.4.1 (2013-11-09)**

* Improvements in filestack creation.
* Updated whitelist for Joomla 3.2

**Version 0.4.0 (2013-11-03)**

* Feature: Automatic signature updates.
* Feature: Submit whitelist requests and malware reports to nekudo.com.
* Feature: Added progress bar.
* Feature: Possibility to run multiple conjobs with different rulesets.
* Bugfix: Problem when running cronjob and web-scan at the same time.
* Multiple minor bugfixes/improvements in source-code.

**Version 0.3.0 (2013-10-06)**

* New feature: Check files for changes by hash-values.
* New feature: Possibility to run Wemahu via cronjob.
* Added basic in-extention help/documentation.
* Performace improvements on machines with little memory.
* Updated regex database.

**Version 0.2.0 (2013-09-23)**

* Improvements in session-storage of reports.
* Added max-result parameters.
* Updated regex database.
* Added whitelist feature.
* Performance improvements while fetching filelist.

**Version 0.1.0 (2013-08-04)**

* First version.

]]>

New blog theme

Sat, 03 Oct 2015 00:00:00 +0000

I'm planning to bring a little more live into this blog again and therefore I needed a new theme. The old one was too dark not suitable for reading longer texts. So I decided to invest a few hours an put together some CSS for a new blog theme.

The new theme is supposed to be simple and optimized for readability - nothing fancy like all the stuff you'll find on themeforest e.g. While choosing the new colors I stumbled across Color Safe which seems to be a nice online tool to create color palettes with focus on readability.

Additionally I decided to publish the source code of this website on GitHub so anyone who is interested may have a look - or even contribute.

]]>

New Project: ShinyGeoip

Sat, 20 Dec 2014 00:00:00 +0000

A few days ago I released a new project called ShinyGeoip.

It is a HTTP API for IP geolocation lookups. You can simply call the API with any valid IPv4 or IPv6 IP address and will get the location information as JSON encoded string. The data is provided by the Maxmind GeoLite2 database.

Here is an example:

http://geoip.nekudo.com/api/87.79.99.25

The API at http://geoip.nekudo.com can be used free of charge but it is also possible to setup your own copy of this API. The sourcecode and install instructions are available on Github under an MIT license.

Feel free to use the API, contribute to the code on github or fork your own project.

]]>

Opensource Projects - Part 2: Taff

Sun, 28 Sep 2014 00:00:00 +0000

Taff is a shortcut for Twitter automatic friend finder.

The sourcecode can be found in this Github repository

At first: This is a really old project - it is at least 4 years old so it most probably won't work out of the box. Nevertheless I wanted to mention it in this series as such script are still in use.

The purpose of this little project was to automatically generate twitter follower for one or more accounts. This mainly happens by following people due to the fact the lots of them will follow you back. Another part of this project was to put some content into the managed twitter accounts. This happens by "recommanding" links.

Technical details

The follower creator cron.scan_twitter.php

This script is responsible for the creation of new followers. I should be periodically called by a cronjob.

Follower creation is done by scanning the twitter public timeline for tweets containig given keywords. If the tweet also is in a matching language the user will be followed. Lots of users will than follow you back given that your account has some interesting tweets in it. If the user does not follow you back within 24 hours he is unfollowed again and will be blacklisted. This ensures the script wont send any new "follow requests" to this user.

The content generator cron.feed_twitter.php

This script should also be executed periodically using a cronjob. I is responsible for creating tweets in your managed twitter accounts.

To generate tweets it uses a given list of RSS feeds and keywords. This feeds are periodically scanned for new articles containig the given keywords. If an article conatains one of the keywords the links is tweeted and "recommanded" to your followers. The keywords used to find articles are the same as the ones which are used to follow users. This ensures the content you recommend is matching the interests of you followers.

Conclusion

As mentioned above - this script is really old and relatively basic. But it gives you a basic idea of how automatic follower creation on twitter works.

Now you know why you got so many new followers the last time you tweeted "iphone" ;)

Got questions? Feel free to create a new issue in the Github repository.

Happy hacking...

]]>

Opensource Projects - Part 1: jitt.li

Wed, 24 Sep 2014 00:00:00 +0000

I'll make it short: Open source is great.

This is why I decided to publish the sourcecode of some projects I did in the past on github. Additonally there will be a blogpost like this one containing some background information regarding the project and the used technology. This way everyone who is interested may learn things, get new ideas or even reuse some of the code.

The first project in this series is: jitt.li (Sourcecode at Github)

What is jitt.li?

jitt.li is a twitterwall. It fetches tweets from the twitter streaming api and displays them in the browser. This happens in realtime using a websocket server.

The first version started as an experiment to test a PHP based webserver I was working on. Later I switched to Ratchet as websocket server.

Technical details

The frontend

The website is based on the Kohana framework. It uses the default bootstrap theme. The interesting part is the jitt.js javascript file. This file contains all the logic of connecting to the websocket server and updating the twitterwall as soon as new tweets arrive.

The backend

The main application running on the server is separated into three parts: The websocket application, a tweet-pull script and a tweet-push script. Additionally there is a simple management-script to start/stop/restart the script on the server.

Websocket application

As mentioned earlier the Ratchet project is used for the websocket server. In the file "jittli_server.php" all the necessary objcts are created and passed to the websocket server which than runs in and endless loop.

The Twitterwall application uses Redis to store information like e.g. the currently active walls so this information can be accessed by the pull and push scripts.

Tweet pull script (sourcecode)

The tweet pull script connects to the twitter steaming api using the phirehose library. It sets the currently active walls (as requested on the website) as filers so only relevant tweets will be received. These tweets are than simply "queued" into a redis list. Every X seconds updates the filters to be set according to the walls currently requested on the website.

Tweet push script (sourcecode)

The push script ready tweets from the redis queue prepares the data and pushes them into the websocket server using a ZeroMQ socket. The websocket server than triggers a method in the Twitterwall application so the tweets can be send to the browser. Here the are handled by the javascript mention in the frontend part.

Summary

When putting it all together this project does the following:

On the jitt.li website visitors can "open" new twitterwalls (stored in redis).
The pull-script uses these walls as filters for the Twitter streaming api. Received tweets are stored into a redis queue.
The push-script reads tweets from the queue and pushes them into the websocket server. (ZeroMQ)
The websocket server triggers the Twitterwall application which sends the data to the browser (using the websocket connection).
The javascript loaded in the browser displays new tweets received from the websocket server.

Got questions? Feel free to open an issue at the Github repository.

]]>