gRPC & Protocol Buffers

What is gRPC ?

gRPC is a modern, open source remote procedure call (RPC) framework that can run anywhere. It is an improved rework of Google’s single purpose RPC infrastructure named Stubby. It uses newest technology standards (i.e SPDY, HTTP/2 and QUIC). You can read more about gRPC principles and requirements at http://www.grpc.io/blog/principles

Why gRPC ?

Pros:

• Static entry point = Reduce HTTP request parsing time
• HTTP/2 Multiplexing (Multiple response/request over a single TCP connection) reduces significant network overhead over HTTP/1.x

Cons:

• As of this writing, no native browsers support yet

What is Protocol Buffers (PB) ?

Protocol buffers is a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more. We define how we want our data to be structured once, then we can use special generated source code to easily write and read our structured data to and from a variety of data streams and using a variety of languages.

PB vs JSON:

• PB is a schema based definition which in a type-safe programming languages it will be checked during compile time. While in a dynamic programming languages (e.g. Javascript, Python, etc) it only reduces boiler plate in de-/serialization.
• PB significantly reduces data on-the-wire size compared to JSON
• PB is harder to debug compared to JSON
• PB is less human readable

Why Protocol Buffers?

One biggest advantage of Protocol Buffers is that is have code generators for multiple language (C++, C#, Go, Java, Python, JavaScript, PHP, Ruby, etc.). The generator will create data structure for each of the language to be used with custom implementation.

Another big advantage is the new fields added in the protocol information will not break any intermediate servers that didn’t need to inspect the data could simply parse it and pass thourgh the data without needing to know about all the fields

gRPC & Protocol Buffers

gRPC by default uses Protocol Buffers as default mechanism for serializing structured data. Combined with HTTP/2, which is a binary protocol, that makes protocol buffers as first choice over serialization format because it is a binary serialization mechanism.

Scala specific implementations:

- Use `scalapb-runtime-grpc` as wrapper for `grpc-java` implementation

- Use `scalapb-compilerplugin` to compile protobuf to scala classes

Some catch during research from other:

1. Connections can die in a somewhat unexpected way. This turned out to be caused by HTTP/2.0 which only allows 1 billion streams over a single connection. Maybe not a common issue, but it hurt us because we had a few processes reaching this limit at the same time, breaking our redundancy. It's easy work around it, and I believe the grpc-java team has plan for a fix that would make this invisible to a single channel.
2. Mixing small/low-latency requests with large/slow requests caused very unstable latency for the low-latency requests. Our current work-around is to start two grpc servers (still within the same java process and sharing the same resources).

Conclusion

While gRPC+Protocol Buffers will increase our API performance, it still lacking in some areas. It is best used for service-to-service communication in microservice architecture and not suitable for browsers usage and/or third-party communication util it gain better supports and has wide support. Although there are some projects to make gRPC usable by browsers (e.g grpc-gateway) it is still in early stages and other research should be done before using them.

Read more!

Javascript Date Formatter with PHP Format String

Every PHP programmer know how to format date using PHP built-in datetime functions or classes and whats the datetime format string to return the date object as date string. But when dealing with javascript I know there are lots of PHP programmer create custom function just to return date object as the function paramter and process the date object to return formatted date string. Thanks to phpjs.org, I created code as bellow to enable date formatting to all javascript Date object using prototype.

Date.prototype.format = function(str) {
    var d = this;
    var s = this.toString(), f = this.getTime();
    return ( '' + str ).replace( /a|A|d|D|F|g|G|h|H|i|I|j|l|L|m|M|n|s|S|t|T|U|w|y|Y|z|Z/g, function( str )
    {
        switch ( str )
        {
            case 'a' : return d.getHours() > 11 ? 'pm' : 'am';
            case 'A' : return d.getHours() > 11 ? 'PM' : 'AM';
            case 'd' : return ( '0' + d.getDate() ).slice(-2);
            case 'D' : return [ 'Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat' ][ d.getDay() ];
            case 'F' : return [ 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December' ][ d.getMonth() ];
            case 'g' : return ( s = ( d.getHours() || 12 ) ) > 12 ? s - 12 : s;
            case 'G' : return d.getHours();
            case 'h' : return ( '0' + ( ( s = d.getHours() || 12 ) > 12 ? s - 12 : s ) ).slice(-2);
            case 'H' : return ( '0' + d.getHours() ).slice(-2);
            case 'i' : return ( '0' + d.getMinutes() ).slice(-2);
            case 'I' : return (function(){ d.setDate(1); d.setMonth(0); s = [ d.getTimezoneOffset() ]; d.setMonth(6); s[1] = d.getTimezoneOffset(); d.setTime( f ); return s[0] == s[1] ? 0 : 1; })();
            case 'j' : return d.getDate();
            case 'l' : return [ 'Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday' ][ d.getDay() ];
            case 'L' : return ( s = d.getFullYear() ) % 4 == 0 && ( s % 100 != 0 || s % 400 == 0 ) ? 1 : 0;
            case 'm' : return ( '0' + ( d.getMonth() + 1 ) ).slice(-2);
            case 'M' : return [ 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec' ][ d.getMonth() ];
            case 'n' : return d.getMonth() + 1;
            case 's' : return ( '0' + d.getSeconds() ).slice(-2);
            case 'S' : return [ 'th', 'st', 'nd', 'rd' ][ ( s = d.getDate() ) < 4 ? s : 0 ];
            case 't' : return (function(){ d.setDate(32); s = 32 - d.getDate(); d.setTime( f ); return s; })();
            case 'T' : return 'UTC';
            case 'U' : return ( '' + f ).slice( 0, -3 );
            case 'w' : return d.getDay();
            case 'y' : return ( '' + d.getFullYear() ).slice(-2);
            case 'Y' : return d.getFullYear();
            case 'z' : return (function(){ d.setMonth(0); return d.setTime( f - d.setDate(1) ) / 86400000; })();
            default : return -d.getTimezoneOffset() * 60;
        }
    });
}

How to use :

var date = new Date('6/2/2013');
console.log(date.format('d-m-Y'));
// Output : 02-06-2013
console.log(date.format('Y-m-d'));
// Output : 2013-06-02
console.log(date.format('Y-m-d H:i:s'));
// Output : 2013-06-02 00:00:00
console.log(date.format('U'));
// Output : 1370106000

Hope that helps (:

Read more!

Vim regex to find whitespaces between quotes

In this example we use the string href="test find white spaces" as an example and we want to encode our URIs with %(percent-encoding) in order to comply with RFC3986.
So we need to encode 'space' character to '%20'. In vim, we can do it simple enough with the following command :
:%s/\(href="[^"]\+\)\@<= /%20/g
been working almost 1 hour to figure this out. :p

Hope that helps somebody. (:

Read more!

LDAP Search Filter

This document outlines how to go about constructing a more sophisticated filter for the userSearchFilter and groupSearchFilter attributes in your AtlassianUser LDAP config file.

Once you have constructed your search filter using this document, you must escape the ampersand symbol and the exclamation mark symbol before adding to your XML file. So for example, (&(objectClass=person)(!(objectClass=user))) becomes (&(objectClass=person)(!(objectClass=user))) Refer to this external documentation on other XML characters that need escaping.

How do I match more than one attribute?

For example, if my users are distinguished by having two objectClass attributes (one equal to 'person' and another to 'user'), this is how I would match for it:

(&(objectClass=person)(objectClass=user))

Notice the ampersand symbol '&' symbol at the start. Translated this means: search for objectClass=person AND object=user.

Alternatively,

(|(objectClass=person)(objectClass=user))

Translated this means: search for objectClass=person OR object=user.

The pipe symbol '|' denotes 'OR'. As this is not a special XML character, then it should not need escaping.

Wildcards

(&(objectClass=user)(cn=*Marketing*))

This means: search for all entries that have objectClass=user AND cn that contains the word 'Marketing'.

How do I match 3 attributes?

This gets a little tricky:

(&(&(objectClass=user)(objectClass=top))(objectClass=person))

Notice how we weave one query into another. For 4 attributes, this would be:

(&(&(&(objectClass=top)(objectClass=person))(objectClass=organizationalPerson))(objectClass=user))

And so on.

Matching Components of Distinguished Names 

You may want to match part of a DN, for instance when you need to look for your groups in two subtrees of your server.

(&(objectClass=group)(|(ou:dn:=Chicago)(ou:dn:=Miami)))

will find groups with an OU component of their DN which is either 'Chicago' or 'Miami'. 

Using 'not'

To exclude entities which match an expression, use '!'. Note that this must be represented as the entity '!' in your XML file.

So

(&(objectClass=group)(&(ou:dn:=Chicago)(!(ou:dn:=Wrigleyville))))

will find all Chicago groups except those with a Wrigleyville OU component.

Note the extra parentheses: (!(<expression>))

Source : http://confluence.atlassian.com/display/DEV/How+to+write+LDAP+search+filters

Read more!

Symfony 2 - User Direct Login / Single Sign On

In applications integration, single sign on is mandatory. In this post, I won't cover how applications integration process. To be more specific, I assume you know how to integrate/share users between multiple applications.

This piece of code bellow will show how to do a login process in symfony 2 programatically for single sign on purpose. But before you can use the following code you have to implement your own custom user class implementing UserInterface for Symfony 2, custom role class implementing RoleInterface and have read Symfony 2 manual about how to setup a firewall in Symfony security configuration file.

The scenario :

We have exisiting web based application called Alice, and I write new application with Symfony 2 called Bob. Now, we want to integrate Bob as part of Alice application. The situation is Alice already have users because it was still used. And Bob will use users from Alice as its users.

The requirements :

Users want to login only one time from Alice login page in order to use both Alice and Bob applications.

The solution :

Flow chart

In above scheme, when a user try to access Bob, it will check for the user's credentials of Alice. If Bob found the user is not authenticated from Alice, it will send a redirect response to Alice login page. If the user is an authenticated user of Alice, then Bob will ask Alice the all the user information it needs and mark the user as an authenticated user in Bob application.

So in Bob authentication validator, we can use the following code to mark the authenticated user of Alice as an authenticated user of Bob.

$token = new UsernamePasswordToken($user, null, $firewall, $user->getRoles());
$this->get('security.context')->setToken($token);
$session = $this->get('session');
$session->set('_security_'.$firewall,serialize($token));

Happy coding (:

Read more!