Unfortunately, I don't think this is supported natively (thought I would love to be shown otherwise!). The way I achieved this was to use the MX Calculator extension to calculate whether a row's count is divisible by 3 or not:
{matrix_test}
<span style="display: none;">
{exp:mx_calc expression="mod({row_count}, 3)"}
</span>
{if '{calc_result}' == '0'}
divisible by 3: {number} <br />
{if:else}
not divisible by 3: {number} <br />
{/if}
{/exp:mx_calc}
{/matrix_test}
This works and you can then customise every third row's outcome, but has a small inconvenience. When you use the exp:mx_calc opening tag to set the {calc_result} variable, it prints out the result of the expression. This prints out:
1 not divisible by 3: 1
not divisible by 3: 2
divisible by 3: 3
not divisible by 3: 4
not divisible by 3: 5
divisible by 3: 6
Unfortunately, I can't use the mx_calc tag right in the conditional, due to parse order issues. The work around is to just wrap the mx_calc tag in a span and give it display: none;
However, I unfortunately skim read your question (which I know I shouldn't do) and I thought you were talking about Grid. In Grid, you can just have mx_calc right in the conditional, because it's not a third party add-on. Thus:
{test}
{if '{exp:mx_calc expression="mod({test:count}, 3)"}' == '0'}
divisible by 3: {test:number} <br />
{if:else}
not divisible by 3: {test:number} <br />
{/if}
{/test}