onsdag den 14. juni 2017

Swift OutputStream Quirk

Writing a zero length array to an OutputStream will effectively close it. The following example demonstrates the issue.



 1
 2
 3
 4
 5
 6
 7
 8
 9
10
let os = OutputStream(toFileAtPath: path, append: false)!
os.open()
print(os.hasSpaceAvailable)
let d = Data(count: 0)
let written = d.withUnsafeBytes { bytes in
    return os.write(bytes, maxLength: d.count)
}
print(os.hasSpaceAvailable)
os.close()
print(os.hasSpaceAvailable)

torsdag den 1. september 2016

Diagnosing SSL errors and enabling TLS 1.2 on Java 7

SSL errors in Java can be tricky to debug if the only thing you got is a stacktrace. However, there is a way to get for information out of the JVM. Enabling the JVM property javax.net.debug will print a lot of debug information. E.g. the version of the SSL/TLS protocol used:

-Djavax.net.debug=all

The default version of TLS on Java 7 is TLS 1.0. The version can changed by using the https.protocols property:

-Dhttps.protocols=TLSv1.2

References

https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https
http://superuser.com/questions/747377/enable-tls-1-1-and-1-2-for-clients-on-java-7

tirsdag den 16. august 2016

Using ssh without polluting your know hosts file

This command will allow you to ssh into a host without adding to the known hosts file:


ssh -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no -i path/to/key ubuntu@12.34.567.890

mandag den 8. august 2016

How to solve Docker maven plugin failure to connect

The following error indicates that your default docker machine, does not exists, has been improperly configured, or is not connected to your shell.


[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.4.3:build (default-cli) on project foobar: Exception caught: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:2375 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused 

The solution depends on which of the three reasons is the case. I usually try this:

eval "$(docker-machine env default)"

If this does not work I would delete the default machine and recreate it:
  1. Delete the default machine.
    docker-machine rm default
    
  2. Recreate the default machine.
    docker-machine create --driver virtualbox default
    
  3. Reconnect the shell to the default machine.
    eval "$(docker-machine env default)"
    

For more information see Docker.com get started.

mandag den 16. maj 2016

Serializing Custom Keys using Jackson

In this post I will show how to serialize a data transfer object (DTO) with a java.util.Map containing a POJO as key to JSON using the Jackson framework in a Jersey web application.

The DTO has a field of type Map with a key of type Foo like this:

import java.net.URI;
import java.util.HashMap;
import java.util.Map;

public class Dto {

    private Map<Foo, URI> fooUris;

    public Dto() {
        this.fooUris = new HashMap<>();
    }

    public ProvisioningDataDto(Map<Foo, URI> fooUris) {
        this.fooUris = fooUris;
    }

    public Map<Foo, URI> getFooUris() {
        return fooUris;
    }

    public void setFooUris(Map<Foo, URI> fooUris) {
        this.fooUris = fooUris;
    }
}

In this example we define Foo to be very simple:

public class Foo {

    private int id;

    public Foo() {
    }

    public Foo(int id) {
        this.id = id;
    }

    public int getId() {
        return this.id;
    }

    public void setId(int id) {
        this.id = id;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Foo foo = (Foo) o;

        return getId() == foo.getId();

    }

    @Override
    public int hashCode() {
        return getId();
    }

}

We will use the Jersey client to send the DTO to the server. The serialization is done using the  the Dto using the javax.ws.rs.client.Entity class:

final Entity<DTO> json = Entity.json(dto);

where dto is DTO object.

Jackson will now complain on not knowing how to deserialize the map. Thus, we need to provide a custom key deserializer. Also the Foo key objects will be serialized by calling the toString method, which is rarely what we would like. Thus I will also show how to provide a key serializer.

First the deserializer. In this case it is very simple, and lacks some checks on the integrity of the data. We simply get the text of the JSON element (line 12) and convert it to an integer (line 13). I leave it as an exercise to the reader to implement proper checks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;

import java.io.IOException;

public class FooDeSerializer extends JsonDeserializer {

    @Override
    public Object deserialize(JsonParser p, DeserializationContext ctxt) throws IOException, JsonProcessingException {
        String str = p.getText().trim();
        return new Foo(Integer.valueOf(str));
    }
}

The serializer is equally simple in this case. We extract the id from the Foo object and use it as the field name of the JSON map element (line 11).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.SerializerProvider;

import java.io.IOException;

public class FooKeySerializer extends JsonSerializer<Foo> {
    @Override
    public void serialize(Foo value, JsonGenerator gen, SerializerProvider serializers) throws IOException, JsonProcessingException {
        gen.writeFieldName(value.getId() + "");
    }
}

The serializers must be enabled in order for Jackson to use them. This is done in two steps. First we add proper annotations to the Foo class. Then we register the serializers to the ObjectMapper used by Jersey.

First we add the annotations @JsonSerialize(keyUsing = FooKeySerializer.class) and
@JsonDeserialize(keyUsing = FooKeyDeSerializer.class) to the Foo class (line 1 and 2):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
@JsonSerialize(keyUsing = FooKeySerializer.class)
@JsonDeserialize(keyUsing = FooKeyDeSerializer.class)
public class Foo {

    private int id;

    public Foo() {
    }

    public Foo(int id) {
        this.id = id;
    }

    public int getId() {
        return this.id;
    }

    public void setId(int id) {
        this.id = id;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Foo foo = (Foo) o;

        return getId() == foo.getId();

    }

    @Override
    public int hashCode() {
        return getId();
    }

}

The we must register the serializers using a Jackson module at the Jersey client:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
  ClientBuilder builder = ClientBuilder.newBuilder();
  builder = builder.register(JacksonFeature.class)
                   .register(new ContextResolver<ObjectMapper>() {

                               @Override
                               public ObjectMapper getContext(Class<?> type) {
                                   ObjectMapper objectMapper = new ObjectMapper();
                                   objectMapper.registerModule(new FooModule());
                                   return objectMapper;
                               }
                           });
        Client jerseyClient = builder.build();

The FooModule simply adds the serializers:

import com.fasterxml.jackson.databind.module.SimpleModule;

public class FooModule extends SimpleModule {

    public FooModule() {
        addKeySerializer(Foo.class, new FooKeySerializer());
        addKeyDeserializer(Foo.class, new FooKeyDeSerializer());
    }
}

Similarly we register the serializers at the server side:

  ResourceConfig packages = resourceConfig.packages("com.example.resources");
  packages.register(JacksonFeature.class)
          .register(new ContextResolver<ObjectMapper>() {

                      @Override
                      public ObjectMapper getContext(Class<?> type) {
                          ObjectMapper objectMapper = new ObjectMapper();
                          objectMapper.registerModule(new FooModule());
                          return objectMapper;
                      }
                  });

Now you will get nice looking JSON like this:

{"fooUris":{"1":"http://example1.com","2":"http:/example2.com","3":"http://example3.com"}}

torsdag den 21. januar 2016

søndag den 13. december 2015

JHead Rename images based on EXIF info

I just discovered JHead, a tool for manipulating JPEG images based on EXIT information. I don't know why I didn't find this tool before. This is extremely useful and convenient. I have tried a couple of tools on Windows and Mac but nothing is as convient at this one.

Install using homebrew:

brew install libjpeg
brew install jhead

libjpeg includes jpgtran, which is prerequisite for jhead but did not install as part of the jhead install.

I name my files using a date timestamp:

jhead -autorot -nf%Y-%m-%d-%Hh%Mm%S *.jpg

References:
http://www.phpied.com/installing-jpegtran-mac-unix-linux/
http://www.sentex.ca/~mwandel/jhead/
http://moinne.com/blog/ronald/bash/rename-your-photos-from-the-comandline-using-jhead

tirsdag den 8. december 2015

Release it

I am reading Release It by Nygard and these are some of the things to remember:


  • Use limits on queries. E.g. LIMIT 1 on Postgres or MySQL
  • Watch out for point-point connections
  • Set Socket.setSoTimeout() to protect against servers under heavy load
  • Use circuit breakers to degrade functionality gracefully under load
  • Return different errors for resource not available and application failures 
  • Fail fast, check availability of all required resources (ram, db, integration points, input)
  • Purge data (database, caches, logs)
  • Steady state
  • Bulkheads 
  • Cooperative demand control, handshaking between client and server to allow clients to back off. Problematic with http as the setup and teardown of connections is often the most expensive. Alternatively consider health checks either from client or from load balancer. Build handshaking into lowlevel protocols. 

SublimeText 2 Shortcuts

I am new to SublimeText 2 and so I need a little reminding of various shortcuts. Multiple cursors is one of the things I loved the most in Emacs. So I find this page very useful.

Simple overview of selection.

torsdag den 26. november 2015

Git Rebase Commands

The goal of this post is to make it easier for me to reference the individual steps of rebasing a Git repository.

I usually push the branch to origin as soon as I have made the first commit for backup purposes. Then I do an interactive rebase when the feature is complete to clean up the branch.

First update develop:


git checkout develop
git pull -u
git checkout feature/branch


Perform the interactive rebase:

git rebase -i develop


Push the feature branch to origin using force. Don't do this if somebody else is working on the same branch.

git push --force


If you don't want to do the rebase interactively, then just rebase:
git rebase develop


git rebase develop

If your current branch contain commits already merged into develop. Or is the continuation of a previous branch merged with develop after the current branch has been created. Then you must apply a little more magic.
Get the hash of the commit immediately before the commits you want to keep. The run the following command with #hash replaced by the hash of the commit.

git rebase --onto master #hash

The changes must be force-pushed.

See more here: https://www.atlassian.com/git/tutorials/merging-vs-rebasing/workflow-walkthrough.

mandag den 16. november 2015

How to list all databases with a given prefix in PostgeSQL

I recently had to delete a large number of databases in a PostgreSql database.

I found the code below very useful. It will list a sequence of SQL DROP DATABASE statements, one for each of the databases with a name starting with "database_name_starting_with_this".

The list of statements can be copied to psql a executed by pasting.

I want to warn you this can be dangerous, so go through the list of statements one by one, before you execute them.


SELECT 'DROP DATABASE "' || datname || '";' FROM pg_database
WHERE datname LIKE 'database_name_starting_with_this%';

fredag den 18. september 2015

How to remove deleted branch names from Git autocomplete


This is actually just a kind of short-cut to this SO. But if you want to skip the reading then here is the one-liner which does the cleaning for you.


git fetch --prune --all


Also to prune all local branches which have already been merged:

git branch -d $(git branch --merged)
git remote prune origin
git branch -d $(git branch | sed '$d')

onsdag den 16. september 2015

How to generate a cryptographically secure random string in JavaScript

When working with security and especially cryptography one often needs to generate a sequence of  bytes which should be uniformly random. This is the case when creating a nonce for OAuth, and many other applications.

This problem is often mistaken for the problem of generating a random string. However, a random sequence of bytes is not a string. But can be represented as a string. And often this is exactly what you want, hence the name of this post.

If I use Google to search for "js random string", then I will get many suggestions on how to generate a random string in JavaScript. However, they are either not cryptographically secure or they rely on Node.js. I will ignore the Node.js versions as they seems to fine.

So what is wrong with the many insecure solutions:
  1. They use Math.random() which is not cryptographically secure.  
  2. They tend to use a reduced alphabet of only alphanumeric characters   

One example of the insecure random is this from this SO:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
function makeid()
{
    var text = "";
    var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";

    for( var i=0; i < 5; i++ )
        text += possible.charAt(Math.floor(Math.random() * possible.length));

    return text;
}


A better approach is to use the window.crypto.getRandomValues method from the WebCrypto API, which states that the Browser is supposed to use a strong strong (pseudo) random number generator. And then encode the result as Base64. Notice that this method will generate a random sequence of numberOfBytes bytes and then base64 encode these. This will generally result in a string of more than numberOfBytes characters.


1
2
3
4
5
function randomString (numberOfBytes) {
    var array = new Uint8Array(length);
    window.crypto.getRandomValues(array);
    return btoa(String.fromCharCode.apply(null, array));
}

This approach is not foolproof either. As mentioned in this SO comment you might not want to trust the client to generate randomness. However, there are situations where you want to, so the call is up to you.

fredag den 6. marts 2015

Sending a WM_COPYDATA from C++ to Python

The following code shows how to send a WM_DATACOPY message from C++ and how to recede it in Python.

The code is based on the code in these posts: http://stackoverflow.com/questions/2451103/use-wm-copydata-to-send-data-between-processes, and http://stackoverflow.com/questions/5249903/receiving-wm-copydata-in-python.

Sending in C++:

1
2
3
4
5
6
7
LPTSTR lpszString = L"Not ready";
COPYDATASTRUCT cds;
cds.dwData = 1;
size_t count_t = sizeof(TCHAR) * (_tcslen(lpszString) + 1);
cds.cbData = static_cast<DWORD>(count_t);
cds.lpData = lpszString;
SendMessage(hwnd, WM_COPYDATA, (WPARAM)hwnd, (LPARAM)(LPVOID)&cds);


Recieving in Python:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
import win32api
import win32process

import struct
import win32con
import win32gui
import array
import win32ui
from ctypes import *


class ACOPYDATASTRUCT(Structure):
    _fields_ = [
        ('dwData', c_ulong),
        ('cbData', c_ulong),
        ('lpData', c_void_p)
    ]
PCOPYDATASTRUCT = POINTER(ACOPYDATASTRUCT)


class Listener:

    def __init__(self):
        message_map = {
            win32con.WM_COPYDATA: self.OnCopyData
        }
        wc = win32gui.WNDCLASS()
        wc.lpfnWndProc = message_map
        wc.lpszClassName = 'MyWindowClass'
        hinst = wc.hInstance = win32api.GetModuleHandle(None)
        classAtom = win32gui.RegisterClass(wc)
        self.hwnd = win32gui.CreateWindow (
            classAtom,
            "win32gui test",
            0,
            0, 
            0,
            win32con.CW_USEDEFAULT, 
            win32con.CW_USEDEFAULT,
            0, 
            0,
            hinst, 
            None
        )
        print self.hwnd

    def OnCopyData(self, hwnd, msg, wparam, lparam):
        print hwnd
        print msg
        print wparam
        print lparam
        pCDS = cast(lparam, PCOPYDATASTRUCT)
        print pCDS.contents.dwData
        print pCDS.contents.cbData
        status = wstring_at(pCDS.contents.lpData)
        print status
        win32gui.PostQuitMessage(0)
        return 1

l = Listener()

win32gui.PumpMessages()


As an additional bonus one can send the hwnd of the receiving python program to the C++ program using a WM_COPYDATA message:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
class COPYDATATYPE(Structure):
    _fields_ = [("cmd",   c_ulong),
                ("hwnd",  c_ulong)]

class COPYDATASTRUCT(Structure):
    _fields_ = [("dwData", c_ulong),
                ("cbData", c_ulong),
                ("lpData", POINTER(COPYDATATYPE))]

cpyData = COPYDATATYPE(1, l.hwnd)
cds = COPYDATASTRUCT(c_ulong(8888),
                     c_ulong(sizeof(cpyData)),
                     pointer(cpyData))

# try to send a message
win32api.SendMessage(target_hwnd,
                     win32con.WM_COPYDATA,
                     0,
                     addressof(cds))


See also my previous post here: http://blog.fagidiot.dk/2015/03/send-wmcopydata-from-python-partial.html

torsdag den 5. marts 2015

Formatting code on this blog

I use http://hilite.me if you have alternative suggestions then don't hesitate to leave a comment.

Send WM_COPYDATA from Python - Partial Solution

I wanted to send a WM_COPYDATA message from a Python application to a Windows application.

Here is the code both Python and C++ which I got work with the only caveat that the string in szData is not received correctly. This example is a very tiny modification of the code posted here: https://groups.google.com/forum/#!topic/comp.lang.python/z3QS7Ic80mU


Python code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import struct
import win32con
import win32gui
import array

class COPYDATATYPE(Structure):
    _fields_ = [("nNum",   c_ulong),
                ("szData", c_char_p)]

class COPYDATASTRUCT(Structure):
    _fields_ = [("dwData", c_ulong),
                ("cbData", c_ulong),
                ("lpData", POINTER(COPYDATATYPE))]

cpyData = COPYDATATYPE(17, '123456789')
cds = COPYDATASTRUCT(c_ulong(1),
                     c_ulong(sizeof(cpyData)),
                     pointer(cpyData))

# try to send a message
win32api.SendMessage(hwnd,
                     win32con.WM_COPYDATA,
                     0,
                     addressof(cds))

C++ code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
 case WM_COPYDATA:
    pCDS = (COPYDATASTRUCT*)lParam;

    if (pCDS->dwData == 1)
    {
      struct COPYDATATYPE
      {
        long nNum;
        char * szData;
      };

      int count = pCDS->cbData;
      PVOID data = pCDS->lpData;
      if (data != NULL)
      {
        COPYDATATYPE *cp = (COPYDATATYPE *)data;
        int i = cp->nNum;
        // The following line does not work.
        const char* message = cp->szData;
      }
    }
    break;

I also found use in reading this: http://www.qsl.net/dl4yhf/yhf_comm/yhf_comm_info.htm#find_window_handle

Get the list of hwnds for a given pid

This Python code shows how to get the list of hwnds from a given pid on Windows.

We use win32gui to enumerate all windows, get their pid to check and then append the hwnd to the list.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
def get_hwnds_for_pid (pid):
  def callback (hwnd, hwnds):
    _, found_pid = win32process.GetWindowThreadProcessId (hwnd)
    if found_pid == pid:
        hwnds.append (hwnd)
    return True
    
  hwnds = []
  win32gui.EnumWindows (callback, hwnds)
  return hwnds

hwnds = get_hwnds_for_pid(3196)
print hwnds

onsdag den 10. december 2014

Debugging path location in Java aka FileNotFoundExceptions

Occasionally I need to write some code which locates stuff on disk using relative paths. And sometime I get the paths wrong :/

It often turns out that the current directory of the application is slightly different than I expect. However, it is not straightforward at first how one determines the current directory of the application.

I have found the following code snippet useful to solve the problem.


1
2
3
4
5
6
import java.nio.file.Path;
import java.nio.file.Paths

Path currentRelativePath = Paths.get("");
String s = currentRelativePath.toAbsolutePath().toString();
System.out.println("Current relative path is: " + s);

It uses NIO so it requires Java 7+.

I got the snippet from this SO.