Help with Chocolate Pots de Creme - Any Pastry Chefs Here?

I made the chocolate pots de creme from Joy of Cooking and they turned out granular even though I kept the temperature below 165 F.  I used Camino's fair trade cocoa and subbed a square of unsweetened chocolate for the 2 semisweet ones the recipe called for.  Any ideas?

6 yolks

2 c milk

1/2 tsp. vanilla

pinch salt

1/2 c sugar

2 oz. (squares) semisweet chocolate

1/4 c. cocoa. 

Here's what I did.  Heated milk in double boiler. Grated fine the chocolate and added when milk hot.  In order: Mixed together yokes, salt, sugar, vanilla, cocoa.  Then added this mix to the hot milk/chocolate mix.  I used a hand held mixer for the mixing.

The final product, while delicious tasting, had tiny brown flecks and a slightly granular texture.  I was hoping for more silky smooth.

Any ideas on what went wrong?

Thanks!